mirror
/
rustc-dev-guide
mirror of https://github.com/rust-lang/rustc-dev-guide.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

[Testing 1/2] Revise testing chapters excluding the directives chapter (#2088)

This commit is contained in:
许杰友 Jieyou Xu (Joe) 2024-10-07 10:39:22 +08:00 committed by GitHub
parent 0a9b996970
commit 2b9274252f
16 changed files with 1385 additions and 1022 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
src/SUMMARY.md
View File

@ -21,10 +21,11 @@
- [Testing with Docker](./tests/docker.md)
- [Testing with CI](./tests/ci.md)
- [Adding new tests](./tests/adding.md)
- [Best practices](./tests/best-practices.md)
- [Compiletest](./tests/compiletest.md)
- [UI tests](./tests/ui.md)
- [Test headers](./tests/headers.md)
- [Integration testing](./tests/integration.md)
- [Test headers](./tests/directives.md)
- [Ecosystem testing](./tests/ecosystem.md)
- [Crater](./tests/crater.md)
- [Fuchsia](./tests/fuchsia.md)
- [Rust for Linux](./tests/rust-for-linux.md)

179
src/tests/adding.md
View File

@ -2,55 +2,62 @@
<!-- toc -->
**In general, we expect every PR that fixes a bug in rustc to come
accompanied by a regression test of some kind.** This test should fail
in master but pass after the PR. These tests are really useful for
preventing us from repeating the mistakes of the past.
**In general, we expect every PR that fixes a bug in rustc to come accompanied
by a regression test of some kind.** This test should fail in master but pass
after the PR. These tests are really useful for preventing us from repeating the
mistakes of the past.
The first thing to decide is which kind of test to add.
This will depend on the nature of the change and what you want to exercise.
Here are some rough guidelines:
The first thing to decide is which kind of test to add. This will depend on the
nature of the change and what you want to exercise. Here are some rough
guidelines:
- The majority of compiler tests are done with [compiletest].
- The majority of compiletest tests are [UI](ui.md) tests in the [`tests/ui`] directory.
- Changes to the standard library are usually tested within the standard library itself.
- The majority of standard library tests are written as doctests,
which illustrate and exercise typical API behavior.
- The majority of compiletest tests are [UI](ui.md) tests in the [`tests/ui`]
directory.
- Changes to the standard library are usually tested within the standard library
itself.
- The majority of standard library tests are written as doctests, which
illustrate and exercise typical API behavior.
- Additional [unit tests](intro.md#package-tests) should go in
`library/${crate}/tests` (where `${crate}` is usually `core`, `alloc`, or `std`).
- If the code is part of an isolated system, and you are not testing compiler output,
consider using a [unit or integration test](intro.md#package-tests).
- Need to run rustdoc? Prefer a `rustdoc` or `rustdoc-ui` test.
Occasionally you'll need `rustdoc-js` as well.
`library/${crate}/tests` (where `${crate}` is usually `core`, `alloc`, or
`std`).
- If the code is part of an isolated system, and you are not testing compiler
output, consider using a [unit or integration test](intro.md#package-tests).
- Need to run rustdoc? Prefer a `rustdoc` or `rustdoc-ui` test. Occasionally
you'll need `rustdoc-js` as well.
- Other compiletest test suites are generally used for special purposes:
- Need to run gdb or lldb? Use the `debuginfo` test suite.
- Need to inspect LLVM IR or MIR IR? Use the `codegen` or `mir-opt` test suites.
- Need to inspect the resulting binary in some way?
Then use `run-make`.
- Need to inspect LLVM IR or MIR IR? Use the `codegen` or `mir-opt` test
suites.
- Need to inspect the resulting binary in some way? Or if all the other test
suites are too limited for your purposes? Then use `run-make`.
- Check out the [compiletest] chapter for more specialized test suites.
After deciding on which kind of test to add, see [best
practices](best-practices.md) for guidance on how to author tests that are easy
to work with that stand the test of time (i.e. if a test fails or need to be
modified several years later, how can we make it easier for them?).
[compiletest]: compiletest.md
[`tests/ui`]: https://github.com/rust-lang/rust/tree/master/tests/ui/
## UI test walkthrough
The following is a basic guide for creating a [UI test](ui.md), which is one
of the most common compiler tests.
For this tutorial, we'll be adding a test for an async error message.
The following is a basic guide for creating a [UI test](ui.md), which is one of
the most common compiler tests. For this tutorial, we'll be adding a test for an
async error message.
### Step 1. Add a test file
### Step 1: Add a test file
The first step is to create a Rust source file somewhere in the
[`tests/ui`] tree.
When creating a test, do your best to find a good location and name (see [Test
organization](ui.md#test-organization) for more).
Since naming is the hardest part of development, everything should be downhill
from here!
The first step is to create a Rust source file somewhere in the [`tests/ui`]
tree. When creating a test, do your best to find a good location and name (see
[Test organization](ui.md#test-organization) for more). Since naming is the
hardest part of development, everything should be downhill from here!
Let's place our async test at `tests/ui/async-await/await-without-async.rs`:
```rust,ignore
// Check what happens when using await in a non-async fn.
// Provide diagnostics when the user writes `await` in a non-`async` function.
//@ edition:2018
async fn foo() {}
@ -64,24 +71,22 @@ fn main() {}
A few things to notice about our test:
* The top should start with a short comment that [explains what the test is
- The top should start with a short comment that [explains what the test is
for](#explanatory_comment).
* The `//@ edition:2018` comment is called a [header](headers.md) which provides
instructions to compiletest on how to build the test.
Here we need to set the edition for `async` to work (the default is 2015).
* Following that is the source of the test.
Try to keep it succinct and to the point.
This may require some effort if you are trying to minimize an example from a
bug report.
* We end this test with an empty `fn main` function.
This is because the default for UI tests is a `bin` crate-type,
and we don't want the "main not found" error in our test.
Alternatively, you could add `#![crate_type="lib"]`.
- The `//@ edition:2018` comment is called a [directive](directives.md) which
provides instructions to compiletest on how to build the test. Here we need to
set the edition for `async` to work (the default is edition 2015).
- Following that is the source of the test. Try to keep it succinct and to the
point. This may require some effort if you are trying to minimize an example
from a bug report.
- We end this test with an empty `fn main` function. This is because the default
for UI tests is a `bin` crate-type, and we don't want the "main not found"
error in our test. Alternatively, you could add `#![crate_type="lib"]`.
### Step 2. Generate the expected output
### Step 2: Generate the expected output
The next step is to create the expected output from the compiler.
This can be done with the `--bless` option:
The next step is to create the expected output snapshots from the compiler. This
can be done with the `--bless` option:
```sh
./x test tests/ui/async-await/await-without-async.rs --bless
@ -91,28 +96,30 @@ This will build the compiler (if it hasn't already been built), compile the
test, and place the output of the compiler in a file called
`tests/ui/async-await/await-without-async.stderr`.
However, this step will fail!
You should see an error message, something like this:
However, this step will fail! You should see an error message, something like
this:
> error: /rust/tests/ui/async-await/await-without-async.rs:7: unexpected
> error: '7:10: 7:16: `await` is only allowed inside `async` functions and
> blocks E0728'
### Step 3. Add error annotations
This is because the stderr contains errors which were not matched by error
annotations in the source file.
Every error needs to be annotated with a comment in the source with the text
of the error.
In this case, we can add the following comment to our test file:
### Step 3: Add error annotations
Every error needs to be annotated with a comment in the source with the text of
the error. In this case, we can add the following comment to our test file:
```rust,ignore
fn bar() {
foo().await
//~^ ERROR `await` is only allowed inside `async` functions and blocks
//~^ ERROR `await` is only allowed inside `async` functions and blocks
}
```
The `//~^` squiggle caret comment tells compiletest that the error belongs to
the previous line (more on this in the [Error
the *previous* line (more on this in the [Error
annotations](ui.md#error-annotations) section).
Save that, and run the test again:
@ -123,16 +130,15 @@ Save that, and run the test again:
It should now pass, yay!
### Step 4. Review the output
### Step 4: Review the output
Somewhat hand-in-hand with the previous step, you should inspect the `.stderr`
file that was created to see if it looks like how you expect.
If you are adding a new diagnostic message, now would be a good time to
also consider how readable the message looks overall, particularly for
people new to Rust.
file that was created to see if it looks like how you expect. If you are adding
a new diagnostic message, now would be a good time to also consider how readable
the message looks overall, particularly for people new to Rust.
Our example `tests/ui/async-await/await-without-async.stderr` file should
look like this:
Our example `tests/ui/async-await/await-without-async.stderr` file should look
like this:
```text
error[E0728]: `await` is only allowed inside `async` functions and blocks
@ -148,48 +154,47 @@ error: aborting due to previous error
For more information about this error, try `rustc --explain E0728`.
```
You may notice some things look a little different than the regular
compiler output.
The `$DIR` removes the path information which will differ between systems.
The `LL` values replace the line numbers.
That helps avoid small changes in the source from triggering large diffs.
See the [Normalization](ui.md#normalization) section for more.
You may notice some things look a little different than the regular compiler
output.
- The `$DIR` removes the path information which will differ between systems.
- The `LL` values replace the line numbers. That helps avoid small changes in
the source from triggering large diffs. See the
[Normalization](ui.md#normalization) section for more.
Around this stage, you may need to iterate over the last few steps a few times
to tweak your test, re-bless the test, and re-review the output.
### Step 5. Check other tests
### Step 5: Check other tests
Sometimes when adding or changing a diagnostic message, this will affect
other tests in the test suite.
The final step before posting a PR is to check if you have affected anything else.
Running the UI suite is usually a good start:
Sometimes when adding or changing a diagnostic message, this will affect other
tests in the test suite. The final step before posting a PR is to check if you
have affected anything else. Running the UI suite is usually a good start:
```sh
./x test tests/ui
```
If other tests start failing, you may need to investigate what has changed
and if the new output makes sense.
If other tests start failing, you may need to investigate what has changed and
if the new output makes sense.
You may also need to re-bless the output with the `--bless` flag.
<a name="explanatory_comment"></a>
## Comment explaining what the test is about
The first comment of a test file should **summarize the point
of the test**, and highlight what is important about it.
If there is an issue number associated with the test, include
the issue number.
The first comment of a test file should **summarize the point of the test**, and
highlight what is important about it. If there is an issue number associated
with the test, include the issue number.
This comment doesn't have to be super extensive. Just something like
"Regression test for #18060: match arms were matching in the wrong
order." might already be enough.
This comment doesn't have to be super extensive. Just something like "Regression
test for #18060: match arms were matching in the wrong order." might already be
enough.
These comments are very useful to others later on when your test
breaks, since they often can highlight what the problem is. They are
also useful if for some reason the tests need to be refactored, since
they let others know which parts of the test were important (often a
test must be rewritten because it no longer tests what is was meant to
test, and then it's useful to know what it *was* meant to test
exactly).
These comments are very useful to others later on when your test breaks, since
they often can highlight what the problem is. They are also useful if for some
reason the tests need to be refactored, since they let others know which parts
of the test were important. Often a test must be rewritten because it no longer
tests what is was meant to test, and then it's useful to know what it *was*
meant to test exactly.

188
src/tests/best-practices.md Normal file
View File

@ -0,0 +1,188 @@
# Best practices for writing tests
This chapter describes best practices related to authoring and modifying tests.
We want to make sure the tests we author are easy to understand and modify, even
several years later, without needing to consult the original author and perform
a bunch of git archeology.
It's good practice to review the test that you authored by pretending that you
are a different contributor who is looking at the test that failed several years
later without much context (this also helps yourself even a few days or months
later!). Then ask yourself: how can I make my life and their lives easier?
To help put this into perspective, let's start with an aside on how to write a
test that makes the life of another contributor as hard as possible.
> **Aside: Simple Test Sabotage Field Manual**
>
> To make the life of another contributor as hard as possible, one might:
>
> - Only name the test after an issue, e.g. `issue-123456.rs`.
> - Have no comments at all on what the test is trying to exercise, no links to
> relevant context.
> - Include a test that is massive (that can otherwise be minimized) and
> contains non-essential pieces which distracts from the core thing the test
> is actually trying to test.
> - Include a bunch of unrelated syntax errors and other errors which are not
> critical to what the test is trying to check.
> - Weirdly format the snippets.
> - Include a bunch of unused and unrelated features.
> - Have e.g. `ignore-windows` [compiletest directives] but don't offer any
> explanation as to *why* they are needed.
## Test naming
Make it easy for the reader to immediately understand what the test is
exercising, instead of having to type in the issue number and dig through github
search for what the test is trying to exercise. This has an additional benefit
of making the test possible to be filtered via `--test-args` as a collection of
related tests.
- Name the test after what it's trying to exercise or prevent regressions of.
- Keep it concise.
- Avoid including issue numbers in test names.
> **Avoid issue numbers in test names**
>
> Prefer including them as links or `#123456` in test comments instead.
>
> ```text
> tests/ui/typeck/issue-123456.rs // bad
> tests/ui/typeck/issue-123456-asm-macro-external-span-ice.rs // bad
> tests/ui/typeck/asm-macro-external-span-ice.rs // good
> ```
>
> `issue-123456.rs` does not tell you immediately anything about what the test
> is actually exercising meaning you need to do additional searching. Including
> the issue number in the test name is really noisy for finding relevant tests
> by what they're exercising (if you `ls` a test directory and get a bunch of
> `issue-xxxxx` prefixes). We can link to the issue in a test comment.
>
> ```rs
> //! Check that `asm!` macro including nested macros that come from external
> //! crates do not lead to a codepoint boundary assertion ICE.
> //!
> //! Regression test for <https://github.com/rust-lang/rust/issues/123456>.
> ```
## Test organization
- For most test suites, try to find a semantically meaningful subdirectory to
home the test.
- E.g. for an implementation of RFC 2093 specifically, we can group a
collection of tests under `tests/ui/rfc-2093-infer-outlives/`. For the
directory name, include what the RFC is about.
- For the [`run-make`] test suite, each `rmake.rs` must be contained within an
immediate subdirectory under `tests/run-make/`. Further nesting is not
presently supported. Avoid including issue number in the directory name too,
include that info in a comment inside `rmake.rs`.
## Test descriptions
To help other contributors understand what the test is about if their changes
lead to the test failing, we should make sure a test has sufficient docs about
its intent/purpose, links to relevant context (incl. issue numbers or other
discussions) and possibly relevant resources (e.g. can be helpful to link to
Win32 APIs for specific behavior).
**Synopsis of a test with good comments**
```rust,ignore
//! Brief summary of what the test is exercising.
//! Example: Regression test for #123456: make sure coverage attribute don't ICE
//! when applied to non-items.
//!
//! Optional: Remarks on related tests/issues, external APIs/tools, crash
//! mechanism, how it's fixed, FIXMEs, limitations, etc.
//! Example: This test is like `tests/attrs/linkage.rs`, but this test is
//! specifically checking `#[coverage]` which exercises a different code
//! path. The ICE was triggered during attribute validation when we tried
//! to construct a `def_path_str` but only emitted the diagnostic when the
//! platform is windows, causing an ICE on unix.
//!
//! Links to relevant issues and discussions. Examples below:
//! Regression test for <https://github.com/rust-lang/rust/issues/123456>.
//! See also <https://github.com/rust-lang/rust/issues/101345>.
//! See discussion at <https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/123456-example-topic>.
//! See [`clone(2)`].
//!
//! [`clone(2)`]: https://man7.org/linux/man-pages/man2/clone.2.html
//@ ignore-windows
// Reason: (why is this test ignored for windows? why not specifically
// windows-gnu or windows-msvc?)
// Optional: Summary of test cases: What positive cases are checked?
// What negative cases are checked? Any specific quirks?
fn main() {
#[coverage]
//~^ ERROR coverage attribute can only be applied to function items.
let _ = {
// Comment highlighting something that deserves reader attention.
fn foo() {}
};
}
```
For how much context/explanation is needed, it is up to the author and
reviewer's discretion. A good rule of thumb is non-trivial things exercised in
the test deserves some explanation to help other contributors to understand.
This may include remarks on:
- How an ICE can get triggered if it's quite elaborate.
- Related issues and tests (e.g. this test is like another test but is kept
separate because...).
- Platform-specific behaviors.
- Behavior of external dependencies and APIs: syscalls, linkers, tools,
environments and the likes.
## Test content
- Try to make sure the test is as minimal as possible.
- Minimize non-critical code and especially minimize unnecessary syntax and type
errors which can clutter stderr snapshots.
- Where possible, use semantically meaningful names (e.g. `fn
bare_coverage_attributes() {}`).
## Flaky tests
All tests need to strive to be reproducible and reliable. Flaky tests are the
worst kind of tests, arguably even worse than not having the test in the first
place.
- Flaky tests can fail in completely unrelated PRs which can confuse other
contributors and waste their time trying to figure out if test failure is
related.
- Flaky tests provide no useful information from its test results other than
it's flaky and not reliable: if a test passed but it's flakey, did I just get
lucky? if a test is flakey but it failed, was it just spurious?
- Flaky tests degrade confidence in the whole test suite. If a test suite can
randomly spuriously fail due to flaky tests, did the whole test suite pass or
did I just get lucky/unlucky?
- Flaky tests can randomly fail in full CI, wasting previous full CI resources.
## Compiletest directives
See [compiletest directives] for a listing of directives.
- For `ignore-*`/`needs-*`/`only-*` directives, unless extremely obvious,
provide a brief remark on why the directive is needed. E.g. `"//@ ignore-wasi
(wasi codegens the main symbol differently)"`.
## FileCheck best practices
See [LLVM FileCheck guide][FileCheck] for details.
- Avoid matching on specific register numbers or basic block numbers unless
they're special or critical for the test. Consider using patterns to match
them where suitable.
> **TODO**
>
> Pending concrete advice.
[compiletest]: ./compiletest.md
[compiletest directives]: ./directives.md
[`run-make`]: ./compiletest.md#run-make-tests
[FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html

347
src/tests/ci.md
View File

@ -1,33 +1,53 @@
# Testing with CI
The primary goal of our CI system is to ensure that the `master` branch of `rust-lang/rust` is always in a valid state and passes our test suite.
The primary goal of our CI system is to ensure that the `master` branch of
`rust-lang/rust` is always in a valid state and passes our test suite.
From a high-level point of view, when you open a pull request at `rust-lang/rust`,
the following will happen:
From a high-level point of view, when you open a pull request at
`rust-lang/rust`, the following will happen:
- A small [subset](#pull-request-builds) of tests and checks are run after each push to the PR. This should help catching common errors.
- A small [subset](#pull-request-builds) of tests and checks are run after each
push to the PR. This should help catching common errors.
- When the PR is approved, the [bors] bot enqueues the PR into a [merge queue].
- Once the PR gets to the front of the queue, bors will create a merge commit
and run the [full test suite](#auto-builds) on it. The merge commit either contains only one specific PR or it can be a ["rollup"](#rollups) which combines multiple PRs together, to save CI costs.
- Once the whole test suite finishes, two things can happen. Either CI fails with an error that needs to be addressed by the developer, or CI succeeds and the merge commit is then pushed to the `master` branch.
and run the [full test suite](#auto-builds) on it. The merge commit either
contains only one specific PR or it can be a ["rollup"](#rollups) which
combines multiple PRs together, to save CI costs.
- Once the whole test suite finishes, two things can happen. Either CI fails
with an error that needs to be addressed by the developer, or CI succeeds and
the merge commit is then pushed to the `master` branch.
If you want to modify what gets executed on CI, see [Modifying CI jobs](#modifying-ci-jobs).
If you want to modify what gets executed on CI, see [Modifying CI
jobs](#modifying-ci-jobs).
## CI workflow
<!-- date-check: may 2024 -->
Our CI is primarily executed on [GitHub Actions], with a single workflow defined in [`.github/workflows/ci.yml`], which contains a bunch of steps that are unified for all CI jobs that we execute. When a commit is pushed to a corresponding branch or a PR, the workflow executes the [`calculate-job-matrix.py`] script, which dynamically generates the specific CI jobs that should be executed. This script uses the [`jobs.yml`] file as an input, which contains a declarative configuration of all our CI jobs.
<!-- date-check: Oct 2024 -->
> Almost all build steps shell out to separate scripts. This keeps the CI fairly platform independent (i.e., we are not overly reliant on GitHub Actions). GitHub Actions is only relied on for bootstrapping the CI process and for orchestrating the scripts that drive the process.
Our CI is primarily executed on [GitHub Actions], with a single workflow defined
in [`.github/workflows/ci.yml`], which contains a bunch of steps that are
unified for all CI jobs that we execute. When a commit is pushed to a
corresponding branch or a PR, the workflow executes the
[`calculate-job-matrix.py`] script, which dynamically generates the specific CI
jobs that should be executed. This script uses the [`jobs.yml`] file as an
input, which contains a declarative configuration of all our CI jobs.
In essence, all CI jobs run `./x test`, `./x dist` or some other command with different configurations, across various operating systems, targets and platforms. There are two broad categories of jobs that are executed, `dist` and non-`dist` jobs.
> Almost all build steps shell out to separate scripts. This keeps the CI fairly
> platform independent (i.e., we are not overly reliant on GitHub Actions).
> GitHub Actions is only relied on for bootstrapping the CI process and for
> orchestrating the scripts that drive the process.
- Dist jobs build a full release of the compiler for a specific platform, including
all the tools we ship through rustup; Those builds are then uploaded to the
`rust-lang-ci2` S3 bucket and are available to be locally installed with the
[rustup-toolchain-install-master] tool; The same builds are also used for
actual releases: our release process basically consists of copying those
artifacts from `rust-lang-ci2` to the production endpoint and signing them.
In essence, all CI jobs run `./x test`, `./x dist` or some other command with
different configurations, across various operating systems, targets and
platforms. There are two broad categories of jobs that are executed, `dist` and
non-`dist` jobs.
- Dist jobs build a full release of the compiler for a specific platform,
including all the tools we ship through rustup; Those builds are then uploaded
to the `rust-lang-ci2` S3 bucket and are available to be locally installed
with the [rustup-toolchain-install-master] tool. The same builds are also used
for actual releases: our release process basically consists of copying those
artifacts from `rust-lang-ci2` to the production endpoint and signing them.
- Non-dist jobs run our full test suite on the platform, and the test suite of
all the tools we ship through rustup; The amount of stuff we test depends on
the platform (for example some tests are run only on Tier 1 platforms), and
@ -37,63 +57,155 @@ artifacts from `rust-lang-ci2` to the production endpoint and signing them.
Based on an input event (usually a push to a branch), we execute one of three
kinds of builds (sets of jobs).
1. PR builds
2. Auto builds
3. Try builds
[rustup-toolchain-install-master]: https://github.com/kennytm/rustup-toolchain-install-master
### Pull Request builds
After each push to a pull request, a set of `pr` jobs are executed. Currently, these execute the
`x86_64-gnu-llvm-X`, `x86_64-gnu-tools`, `mingw-check` and `mingw-check-tidy` jobs, all running on Linux. These execute a relatively short (~30 minutes) and lightweight test suite that should catch common issues. More specifically, they run a set of lints, they try to perform a cross-compile check build to Windows mingw (without producing any artifacts) and they test the compiler using a *system* version of LLVM. Unfortunately, it would take too many resources to run the full test suite for each commit on every PR.
After each push to a pull request, a set of `pr` jobs are executed. Currently,
these execute the `x86_64-gnu-llvm-X`, `x86_64-gnu-tools`, `mingw-check` and
`mingw-check-tidy` jobs, all running on Linux. These execute a relatively short
(~30 minutes) and lightweight test suite that should catch common issues. More
specifically, they run a set of lints, they try to perform a cross-compile check
build to Windows mingw (without producing any artifacts) and they test the
compiler using a *system* version of LLVM. Unfortunately, it would take too many
resources to run the full test suite for each commit on every PR.
PR jobs are defined in the `pr` section of [`jobs.yml`]. They run under the `rust-lang/rust` repository, and their results can be observed directly on the PR, in the "CI checks" section at the bottom of the PR page.
> **Note on doc comments**
>
> Note that PR CI as of Oct 2024 <!-- datecheck --> by default does not try to
> run `./x doc xxx`. This means that if you have any broken intradoc links that
> would lead to `./x doc xxx` failing, it will happen very late into the full
> merge queue CI pipeline.
>
> Thus, it is a good idea to run `./x doc xxx` locally for any doc comment
> changes to help catch these early.
PR jobs are defined in the `pr` section of [`jobs.yml`]. They run under the
`rust-lang/rust` repository, and their results can be observed directly on the
PR, in the "CI checks" section at the bottom of the PR page.
### Auto builds
Before a commit can be merged into the `master` branch, it needs to pass our complete test suite. We call this an `auto` build. This build runs tens of CI jobs that exercise various tests across operating systems and targets.
The full test suite is quite slow; it can take two hours or more until all the `auto` CI jobs finish.
Before a commit can be merged into the `master` branch, it needs to pass our
complete test suite. We call this an `auto` build. This build runs tens of CI
jobs that exercise various tests across operating systems and targets. The full
test suite is quite slow; it can take two hours or more until all the `auto` CI
jobs finish.
Most platforms only run the build steps, some run a restricted set of tests, only a subset run the full suite of tests (see Rust's [platform tiers]).
Most platforms only run the build steps, some run a restricted set of tests,
only a subset run the full suite of tests (see Rust's [platform tiers]).
Auto jobs are defined in the `auto` section of [`jobs.yml`]. They are executed on the `auto` branch under the `rust-lang-ci/rust` repository[^rust-lang-ci] and their results can be seen [here](https://github.com/rust-lang-ci/rust/actions), although usually you will be notified of the result by a comment made by bors on the corresponding PR.
Auto jobs are defined in the `auto` section of [`jobs.yml`]. They are executed
on the `auto` branch under the `rust-lang-ci/rust` repository[^rust-lang-ci] and
their results can be seen [here](https://github.com/rust-lang-ci/rust/actions),
although usually you will be notified of the result by a comment made by bors on
the corresponding PR.
At any given time, at most a single `auto` build is being executed. Find out more [here](#merging-prs-serially-with-bors).
At any given time, at most a single `auto` build is being executed. Find out
more [here](#merging-prs-serially-with-bors).
[platform tiers]: https://forge.rust-lang.org/release/platform-support.html#rust-platform-support
[^rust-lang-ci]: The `auto` and `try` jobs run under the `rust-lang-ci` fork for historical reasons. This may change in the future.
[^rust-lang-ci]: The `auto` and `try` jobs run under the `rust-lang-ci` fork for
historical reasons. This may change in the future.
### Try builds
Sometimes we want to run a subset of the test suite on CI for a given PR, or build a set of compiler artifacts from that PR, without attempting to merge it. We call this a "try build". A try build is started after a user with the proper permissions posts a PR comment with the `@bors try` command.
Sometimes we want to run a subset of the test suite on CI for a given PR, or
build a set of compiler artifacts from that PR, without attempting to merge it.
We call this a "try build". A try build is started after a user with the proper
permissions posts a PR comment with the `@bors try` command.
There are several use-cases for try builds:
- Run a set of performance benchmarks using our [rustc-perf] benchmark suite. For this, a working compiler build is needed, which can be generated with a try build that runs the [dist-x86_64-linux] CI job, which builds an optimized version of the compiler on Linux (this job is currently executed by default when you start a try build). To create a try build and schedule it for a performance benchmark, you can use the `@bors try @rust-timer queue` command combination.
- Check the impact of the PR across the Rust ecosystem, using a [crater] run. Again, a working compiler build is needed for this, which can be produced by the [dist-x86_64-linux] CI job.
- Run a specific CI job (e.g. Windows tests) on a PR, to quickly test if it passes the test suite executed by that job. You can select which CI jobs will be executed in the try build by adding up to 10 lines containing `try-job: <name of job>` to the PR description. All such specified jobs will be executed in the try build once the `@bors try` command is used on the PR. If no try jobs are specified in this way, the jobs defined in the `try` section of [`jobs.yml`] will be executed by default.
- Run a set of performance benchmarks using our [rustc-perf] benchmark suite.
For this, a working compiler build is needed, which can be generated with a
try build that runs the [dist-x86_64-linux] CI job, which builds an optimized
version of the compiler on Linux (this job is currently executed by default
when you start a try build). To create a try build and schedule it for a
performance benchmark, you can use the `@bors try @rust-timer queue` command
combination.
- Check the impact of the PR across the Rust ecosystem, using a [crater] run.
Again, a working compiler build is needed for this, which can be produced by
the [dist-x86_64-linux] CI job.
- Run a specific CI job (e.g. Windows tests) on a PR, to quickly test if it
passes the test suite executed by that job. You can select which CI jobs will
be executed in the try build by adding up to 10 lines containing `try-job:
<name of job>` to the PR description. All such specified jobs will be executed
in the try build once the `@bors try` command is used on the PR. If no try
jobs are specified in this way, the jobs defined in the `try` section of
[`jobs.yml`] will be executed by default.
Try jobs are defined in the `try` section of [`jobs.yml`]. They are executed on the `try` branch under the `rust-lang-ci/rust` repository[^rust-lang-ci] and their results can be seen [here](https://github.com/rust-lang-ci/rust/actions), although usually you will be notified of the result by a comment made by bors on the corresponding PR.
> **Using `try-job` PR description directives**
>
> 1. Identify which set of try-jobs (max 10) you would like to exercise. You can
> find the name of the CI jobs in [`jobs.yml`].
>
> 2. Amend PR description to include (usually at the end of the PR description)
> e.g.
>
> ```text
> This PR fixes #123456.
>
> try-job: x86_64-msvc
> try-job: test-various
> ```
>
> Each `try-job` directive must be on its own line.
>
> 3. Run the prescribed try jobs with `@bors try`. As aforementioned, this
> requires the user to either (1) have `try` permissions or (2) be delegated
> with `try` permissions by `@bors delegate` by someone who has `try`
> permissions.
>
> Note that this is usually easier to do than manually edit [`jobs.yml`].
> However, it can be less flexible because you cannot adjust the set of tests
> that are exercised this way.
Multiple try builds can execute concurrently across different PRs, but inside a single PR, at most one try build can be executing at the same time.
Try jobs are defined in the `try` section of [`jobs.yml`]. They are executed on
the `try` branch under the `rust-lang-ci/rust` repository[^rust-lang-ci] and
their results can be seen [here](https://github.com/rust-lang-ci/rust/actions),
although usually you will be notified of the result by a comment made by bors on
the corresponding PR.
Multiple try builds can execute concurrently across different PRs.
<div class="warning">
bors identify try jobs by commit hash. This means that if you have two PRs
containing the same (latest) commits, running `@bors try` will result in the
*same* try job and it really confuses `bors`. Please refrain from doing so.
</div>
[rustc-perf]: https://github.com/rust-lang/rustc-perf
[crater]: https://github.com/rust-lang/crater
### Modifying CI jobs
If you want to modify what gets executed on our CI, you can simply modify the `pr`, `auto` or `try` sections of the [`jobs.yml`] file.
If you want to modify what gets executed on our CI, you can simply modify the
`pr`, `auto` or `try` sections of the [`jobs.yml`] file.
You can also modify what gets executed temporarily, for example to test a particular platform
or configuration that is challenging to test locally (for example, if a Windows build fails, but you don't have access to a Windows machine). Don't hesitate to use CI resources in such situations to try out a fix!
You can also modify what gets executed temporarily, for example to test a
particular platform or configuration that is challenging to test locally (for
example, if a Windows build fails, but you don't have access to a Windows
machine). Don't hesitate to use CI resources in such situations to try out a
fix!
You can perform an arbitrary CI job in two ways:
- Use the [try build](#try-builds) functionality, and specify the CI jobs that you want to be
executed in try builds in your PR description.
- Modify the [`pr`](#pull-request-builds) section of `jobs.yml` to specify which CI jobs should be
executed after each push to your PR. This might be faster than repeatedly starting try builds.
- Use the [try build](#try-builds) functionality, and specify the CI jobs that
you want to be executed in try builds in your PR description.
- Modify the [`pr`](#pull-request-builds) section of `jobs.yml` to specify which
CI jobs should be executed after each push to your PR. This might be faster
than repeatedly starting try builds.
To modify the jobs executed after each push to a PR, you can simply copy one of the job definitions from the `auto` section to the `pr` section. For example, the `x86_64-msvc` job is responsible for running the 64-bit MSVC tests.
You can copy it to the `pr` section to cause it to be executed after a commit is pushed to your
PR, like this:
To modify the jobs executed after each push to a PR, you can simply copy one of
the job definitions from the `auto` section to the `pr` section. For example,
the `x86_64-msvc` job is responsible for running the 64-bit MSVC tests. You can
copy it to the `pr` section to cause it to be executed after a commit is pushed
to your PR, like this:
```yaml
pr:
@ -109,61 +221,69 @@ pr:
<<: *job-windows-8c
```
Then you can commit the file and push it to your PR branch on GitHub. GitHub Actions should then
execute this CI job after each push to your PR.
Then you can commit the file and push it to your PR branch on GitHub. GitHub
Actions should then execute this CI job after each push to your PR.
**After you have finished your experiments, don't forget to remove any changes you have made to `jobs.yml`, if they were supposed to be temporary!**
<div class="warning">
**After you have finished your experiments, don't forget to remove any changes
you have made to `jobs.yml`, if they were supposed to be temporary!**
A good practice is to prefix `[WIP]` in PR title while still running try jobs
and `[DO NOT MERGE]` in the commit that modifies the CI jobs for testing
purposes.
</div>
Although you are welcome to use CI, just be conscious that this is a shared
resource with limited concurrency. Try not to enable too many jobs at once (one or two should be sufficient in
most cases).
resource with limited concurrency. Try not to enable too many jobs at once (one
or two should be sufficient in most cases).
## Merging PRs serially with bors
CI services usually test the last commit of a branch merged with the last
commit in `master`, and while that’s great to check if the feature works in
isolation, it doesn’t provide any guarantee the code is going to work once it’s
merged. Breakages like these usually happen when another, incompatible PR is
merged after the build happened.
CI services usually test the last commit of a branch merged with the last commit
in `master`, and while that’s great to check if the feature works in isolation,
it doesn’t provide any guarantee the code is going to work once it’s merged.
Breakages like these usually happen when another, incompatible PR is merged
after the build happened.
To ensure a `master` branch that works all the time, we forbid manual merges. Instead,
all PRs have to be approved through our bot, [bors] (the software behind it is
called [homu]). All the approved PRs are put [in a queue][merge queue] (sorted
by priority and creation date) and are automatically tested one at the time. If
all the builders are green, the PR is merged, otherwise the failure is recorded
and the PR will have to be re-approved again.
To ensure a `master` branch that works all the time, we forbid manual merges.
Instead, all PRs have to be approved through our bot, [bors] (the software
behind it is called [homu]). All the approved PRs are put in a [merge queue]
(sorted by priority and creation date) and are automatically tested one at the
time. If all the builders are green, the PR is merged, otherwise the failure is
recorded and the PR will have to be re-approved again.
Bors doesn’t interact with CI services directly, but it works by pushing the
merge commit it wants to test to specific branches (like `auto` or `try`), which
are configured to execute CI checks. Bors then detects the
outcome of the build by listening for either Commit Statuses or Check Runs.
Since the merge commit is based on the latest `master` and only one can be tested
at the same time, when the results are green, `master` is fast-forwarded to that
merge commit.
are configured to execute CI checks. Bors then detects the outcome of the build
by listening for either Commit Statuses or Check Runs. Since the merge commit is
based on the latest `master` and only one can be tested at the same time, when
the results are green, `master` is fast-forwarded to that merge commit.
Unfortunately testing a single PR at the time, combined with our long CI (~2
hours for a full run), means we can’t merge too many PRs in a single day, and a
single failure greatly impacts our throughput for the day. The maximum number
of PRs we can merge in a day is around ~10.
single failure greatly impacts our throughput for the day. The maximum number of
PRs we can merge in a day is around ~10.
The large CI run times and requirement for a large builder pool is largely due to the
fact that full release artifacts are built in the `dist-` builders. This is worth it
because these release artifacts:
The large CI run times and requirement for a large builder pool is largely due
to the fact that full release artifacts are built in the `dist-` builders. This
is worth it because these release artifacts:
- Allow perf testing even at a later date
- Allow bisection when bugs are discovered later
- Ensure release quality since if we're always releasing, we can catch problems early
- Allow perf testing even at a later date.
- Allow bisection when bugs are discovered later.
- Ensure release quality since if we're always releasing, we can catch problems
early.
### Rollups
Some PRs don’t need the full test suite to be executed: trivial changes like
typo fixes or README improvements *shouldn’t* break the build, and testing
every single one of them for 2+ hours is a big waste of time. To solve this,
we regularly create a "rollup", a PR where we merge several pending trivial
PRs so they can be tested together. Rollups are created manually by a team member
using the "create a rollup" button on the [merge queue]. The team member uses their
judgment to decide if a PR is risky or not, and are the best tool we have at
the moment to keep the queue in a manageable state.
typo fixes or README improvements *shouldn’t* break the build, and testing every
single one of them for 2+ hours is a big waste of time. To solve this, we
regularly create a "rollup", a PR where we merge several pending trivial PRs so
they can be tested together. Rollups are created manually by a team member using
the "create a rollup" button on the [merge queue]. The team member uses their
judgment to decide if a PR is risky or not, and are the best tool we have at the
moment to keep the queue in a manageable state.
## Docker
@ -174,20 +294,25 @@ platform’s custom [Docker container]. This has a lot of advantages for us:
underlying image (switching from the trusty image to xenial was painless for
us).
- We can use ancient build environments to ensure maximum binary compatibility,
for example [using older CentOS releases][dist-x86_64-linux] on our Linux builders.
- We can avoid reinstalling tools (like QEMU or the Android emulator) every
time thanks to Docker image caching.
for example [using older CentOS releases][dist-x86_64-linux] on our Linux
builders.
- We can avoid reinstalling tools (like QEMU or the Android emulator) every time
thanks to Docker image caching.
- Users can run the same tests in the same environment locally by just running
`src/ci/docker/run.sh image-name`, which is awesome to debug failures.
`src/ci/docker/run.sh image-name`, which is awesome to debug failures. Note
that there are only linux docker images available locally due to licensing and
other restrictions.
The docker images prefixed with `dist-` are used for building artifacts while those without that prefix run tests and checks.
The docker images prefixed with `dist-` are used for building artifacts while
those without that prefix run tests and checks.
We also run tests for less common architectures (mainly Tier 2 and Tier 3
platforms) in CI. Since those platforms are not x86 we either run
everything inside QEMU or just cross-compile if we don’t want to run the tests
for that platform.
platforms) in CI. Since those platforms are not x86 we either run everything
inside QEMU or just cross-compile if we don’t want to run the tests for that
platform.
These builders are running on a special pool of builders set up and maintained for us by GitHub.
These builders are running on a special pool of builders set up and maintained
for us by GitHub.
[Docker container]: https://github.com/rust-lang/rust/tree/master/src/ci/docker
@ -198,10 +323,11 @@ Our CI workflow uses various caching mechanisms, mainly for two things:
### Docker images caching
The Docker images we use to run most of the Linux-based builders take a *long*
time to fully build. To speed up the build, we cache it using [Docker registry caching],
with the intermediate artifacts being stored on [ghcr.io]. We also push the built
Docker images to ghcr, so that they can be reused by other tools (rustup) or
by developers running the Docker build locally (to speed up their build).
time to fully build. To speed up the build, we cache it using [Docker registry
caching], with the intermediate artifacts being stored on [ghcr.io]. We also
push the built Docker images to ghcr, so that they can be reused by other tools
(rustup) or by developers running the Docker build locally (to speed up their
build).
Since we test multiple, diverged branches (`master`, `beta` and `stable`), we
can’t rely on a single cache for the images, otherwise builds on a branch would
@ -216,8 +342,9 @@ Dockerfiles and related scripts.
We build some C/C++ stuff in various CI jobs, and we rely on [sccache] to cache
the intermediate LLVM artifacts. Sccache is a distributed ccache developed by
Mozilla, which can use an object storage bucket as the storage backend. In our case,
the artefacts are uploaded to an S3 bucket that we control (`rust-lang-ci-sccache2`).
Mozilla, which can use an object storage bucket as the storage backend. In our
case, the artefacts are uploaded to an S3 bucket that we control
(`rust-lang-ci-sccache2`).
[sccache]: https://github.com/mozilla/sccache
@ -228,16 +355,16 @@ During the years we developed some custom tooling to improve our CI experience.
### Rust Log Analyzer to show the error message in PRs
The build logs for `rust-lang/rust` are huge, and it’s not practical to find
what caused the build to fail by looking at the logs. To improve the
developers’ experience we developed a bot called [Rust Log Analyzer][rla] (RLA)
that receives the build logs on failure and extracts the error message
automatically, posting it on the PR.
what caused the build to fail by looking at the logs. To improve the developers’
experience we developed a bot called [Rust Log Analyzer][rla] (RLA) that
receives the build logs on failure and extracts the error message automatically,
posting it on the PR.
The bot is not hardcoded to look for error strings, but was trained with a
bunch of build failures to recognize which lines are common between builds and
which are not. While the generated snippets can be weird sometimes, the bot is
pretty good at identifying the relevant lines even if it’s an error we've never
seen before.
The bot is not hardcoded to look for error strings, but was trained with a bunch
of build failures to recognize which lines are common between builds and which
are not. While the generated snippets can be weird sometimes, the bot is pretty
good at identifying the relevant lines even if it’s an error we've never seen
before.
[rla]: https://github.com/rust-lang/rust-log-analyzer
@ -245,16 +372,16 @@ seen before.
The `rust-lang/rust` repo doesn’t only test the compiler on its CI, but also a
variety of tools and documentation. Some documentation is pulled in via git
submodules. If we blocked merging rustc PRs on the documentation being fixed,
we would be stuck in a chicken-and-egg problem, because the documentation's CI
would not pass since updating it would need the not-yet-merged version of
rustc to test against (and we usually require CI to be passing).
submodules. If we blocked merging rustc PRs on the documentation being fixed, we
would be stuck in a chicken-and-egg problem, because the documentation's CI
would not pass since updating it would need the not-yet-merged version of rustc
to test against (and we usually require CI to be passing).
To avoid the problem, submodules are allowed to fail, and their status is
recorded in [rust-toolstate]. When a submodule breaks, a bot automatically
pings the maintainers so they know about the breakage, and it records the
failure on the toolstate repository. The release process will then ignore
broken tools on nightly, removing them from the shipped nightlies.
recorded in [rust-toolstate]. When a submodule breaks, a bot automatically pings
the maintainers so they know about the breakage, and it records the failure on
the toolstate repository. The release process will then ignore broken tools on
nightly, removing them from the shipped nightlies.
While tool failures are allowed most of the time, they’re automatically
forbidden a week before a release: we don’t care if tools are broken on nightly

604
src/tests/compiletest.md
View File

@ -4,158 +4,163 @@
## Introduction
`compiletest` is the main test harness of the Rust test suite.
It allows test authors to organize large numbers of tests
(the Rust compiler has many thousands),
efficient test execution (parallel execution is supported),
and allows the test author to configure behavior and expected results of both
`compiletest` is the main test harness of the Rust test suite. It allows test
authors to organize large numbers of tests (the Rust compiler has many
thousands), efficient test execution (parallel execution is supported), and
allows the test author to configure behavior and expected results of both
individual and groups of tests.
> NOTE:
> For macOS users, `SIP` (System Integrity Protection) [may consistently
> check the compiled binary by sending network requests to Apple][zulip],
> so you may get a huge performance degradation when running tests.
> **Note for macOS users**
>
> You can resolve it by tweaking the following settings:
> `Privacy & Security -> Developer Tools -> Add Terminal (Or VsCode, etc.)`.
> For macOS users, `SIP` (System Integrity Protection) [may consistently check
> the compiled binary by sending network requests to Apple][zulip], so you may
> get a huge performance degradation when running tests.
>
> You can resolve it by tweaking the following settings: `Privacy & Security ->
> Developer Tools -> Add Terminal (Or VsCode, etc.)`.
[zulip]: https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/.E2.9C.94.20Is.20there.20any.20performance.20issue.20for.20MacOS.3F
`compiletest` may check test code for success, for runtime failure,
or for compile-time failure.
Tests are typically organized as a Rust source file with annotations in
comments before and/or within the test code.
These comments serve to direct `compiletest` on if or how to run the test,
what behavior to expect, and more.
See [header commands](headers.md) and the test suite documentation below
for more details on these annotations.
`compiletest` may check test code for compile-time or run-time success/failure.
See the [Adding new tests](adding.md) chapter for a tutorial on creating a new
test, and the [Running tests](running.md) chapter on how to run the test
suite.
Tests are typically organized as a Rust source file with annotations in comments
before and/or within the test code. These comments serve to direct `compiletest`
on if or how to run the test, what behavior to expect, and more. See
[directives](directives.md) and the test suite documentation below for more details
on these annotations.
Compiletest itself tries to avoid running tests when the artifacts
that are involved (mainly the compiler) haven't changed. You can use
`x test --test-args --force-rerun` to rerun a test even when none of the
inputs have changed.
See the [Adding new tests](adding.md) and [Best practies](best-practiecs.md)
chapters for a tutorial on creating a new test and advice on writing a good
test, and the [Running tests](running.md) chapter on how to run the test suite.
Compiletest itself tries to avoid running tests when the artifacts that are
involved (mainly the compiler) haven't changed. You can use `x test --test-args
--force-rerun` to rerun a test even when none of the inputs have changed.
## Test suites
All of the tests are in the [`tests`] directory.
The tests are organized into "suites", with each suite in a separate subdirectory.
Each test suite behaves a little differently, with different compiler behavior
and different checks for correctness.
For example, the [`tests/incremental`] directory contains tests for
incremental compilation.
The various suites are defined in [`src/tools/compiletest/src/common.rs`] in
the `pub enum Mode` declaration.
All of the tests are in the [`tests`] directory. The tests are organized into
"suites", with each suite in a separate subdirectory. Each test suite behaves a
little differently, with different compiler behavior and different checks for
correctness. For example, the [`tests/incremental`] directory contains tests for
incremental compilation. The various suites are defined in
[`src/tools/compiletest/src/common.rs`] in the `pub enum Mode` declaration.
The following test suites are available, with links for more information:
- [`ui`](ui.md) — tests that check the stdout/stderr from the compilation
and/or running the resulting executable
- `ui-fulldeps` — `ui` tests which require a linkable build of `rustc` (such
as using `extern crate rustc_span;` or used as a plugin)
- [`pretty`](#pretty-printer-tests) — tests for pretty printing
- [`incremental`](#incremental-tests) — tests incremental compilation behavior
- [`debuginfo`](#debuginfo-tests) — tests for debuginfo generation running debuggers
- [`codegen`](#codegen-tests) — tests for code generation
- [`codegen-units`](#codegen-units-tests) — tests for codegen unit partitioning
- [`assembly`](#assembly-tests) — verifies assembly output
- [`mir-opt`](#mir-opt-tests) — tests for MIR generation
- [`run-make`](#run-make-tests) — general purpose tests using Rust programs (or
Makefiles (legacy))
- [`run-pass-valgrind`](#valgrind-tests) — tests run with Valgrind
- [`coverage`](#coverage-tests) - tests for coverage instrumentation
- [`coverage-run-rustdoc`](#coverage-tests) - coverage tests that also run
instrumented doctests
- [Rustdoc tests](../rustdoc.md#tests):
- `rustdoc` — tests for rustdoc, making sure that the generated files
contain the expected documentation.
- `rustdoc-gui` — tests for rustdoc's GUI using a web browser.
- `rustdoc-js` — tests to ensure the rustdoc search is working as expected.
- `rustdoc-js-std` — tests to ensure the rustdoc search is working as expected
(run specifically on the std docs).
- `rustdoc-json` — tests on the JSON output of rustdoc.
- `rustdoc-ui` — tests on the terminal output of rustdoc.
### Compiler-specific test suites
| Test suite | Purpose |
|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
| [`ui`](ui.md) | Check the stdout/stderr snapshots from the compilation and/or running the resulting executable |
| `ui-fulldeps` | `ui` tests which require a linkable build of `rustc` (such as using `extern crate rustc_span;` or used as a plugin) |
| [`pretty`](#pretty-printer-tests) | Check pretty printing |
| [`incremental`](#incremental-tests) | Check incremental compilation behavior |
| [`debuginfo`](#debuginfo-tests) | Check debuginfo generation running debuggers |
| [`codegen`](#codegen-tests) | Check code generation |
| [`codegen-units`](#codegen-units-tests) | Check codegen unit partitioning |
| [`assembly`](#assembly-tests) | Check assembly output |
| [`mir-opt`](#mir-opt-tests) | Check MIR generation and optimizations |
| [`run-pass-valgrind`](#valgrind-tests) | Run with Valgrind |
| [`coverage`](#coverage-tests) | Check coverage instrumentation |
| [`coverage-run-rustdoc`](#coverage-tests) | `coverage` tests that also run instrumented doctests |
### General purpose test suite
[`run-make`](#run-make-tests) are general purpose tests using Rust programs (or
Makefiles (legacy)).
### Rustdoc test suites
See [Rustdoc tests](../rustdoc.md#tests) for more details.
| Test suite | Purpose |
|------------------|--------------------------------------------------------------------------|
| `rustdoc` | Check `rustdoc` generated files contain the expected documentation |
| `rustdoc-gui` | Check `rustdoc`'s GUI using a web browser |
| `rustdoc-js` | Check `rustdoc` search is working as expected |
| `rustdoc-js-std` | Check rustdoc search is working as expected specifically on the std docs |
| `rustdoc-json` | Check JSON output of `rustdoc` |
| `rustdoc-ui` | Check terminal output of `rustdoc` |
[`tests`]: https://github.com/rust-lang/rust/blob/master/tests
[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
### Pretty-printer tests
The tests in [`tests/pretty`] exercise the "pretty-printing" functionality of `rustc`.
The `-Z unpretty` CLI option for `rustc` causes it to translate the input source
into various different formats, such as the Rust source after macro expansion.
The tests in [`tests/pretty`] exercise the "pretty-printing" functionality of
`rustc`. The `-Z unpretty` CLI option for `rustc` causes it to translate the
input source into various different formats, such as the Rust source after macro
expansion.
The pretty-printer tests have several [header commands](headers.md) described below.
The pretty-printer tests have several [directives](directives.md) described below.
These commands can significantly change the behavior of the test, but the
default behavior without any commands is to:
1. Run `rustc -Zunpretty=normal` on the source file
2. Run `rustc -Zunpretty=normal` on the output of the previous step
1. Run `rustc -Zunpretty=normal` on the source file.
2. Run `rustc -Zunpretty=normal` on the output of the previous step.
3. The output of the previous two steps should be the same.
4. Run `rustc -Zno-codegen` on the output to make sure that it can type check
(this is similar to running `cargo check`)
(this is similar to running `cargo check`).
If any of the commands above fail, then the test fails.
The header commands for pretty-printing tests are:
The directives for pretty-printing tests are:
* `pretty-mode` specifies the mode pretty-print tests should run in
(that is, the argument to `-Zunpretty`).
The default is `normal` if not specified.
* `pretty-compare-only` causes a pretty test to only compare the pretty-printed output
(stopping after step 3 from above).
It will not try to compile the expanded output to type check it.
This is needed for a pretty-mode that does not expand to valid
Rust, or for other situations where the expanded output cannot be compiled.
* `pretty-expanded` allows a pretty test to also check that the expanded
output can be type checked.
That is, after the steps above, it does two more steps:
- `pretty-mode` specifies the mode pretty-print tests should run in (that is,
the argument to `-Zunpretty`). The default is `normal` if not specified.
- `pretty-compare-only` causes a pretty test to only compare the pretty-printed
output (stopping after step 3 from above). It will not try to compile the
expanded output to type check it. This is needed for a pretty-mode that does
not expand to valid Rust, or for other situations where the expanded output
cannot be compiled.
- `pretty-expanded` allows a pretty test to also check that the expanded output
can be type checked. That is, after the steps above, it does two more steps:
> 5. Run `rustc -Zunpretty=expanded` on the original source
> 6. Run `rustc -Zno-codegen` on the expanded output to make sure that it can type check
This is needed because not all code can be compiled after being expanded.
Pretty tests should specify this if they can.
An example where this cannot be used is if the test includes `println!`.
That macro expands to reference private internal functions of the standard
library that cannot be called directly without the `fmt_internals` feature
gate.
Pretty tests should specify this if they can. An example where this cannot be
used is if the test includes `println!`. That macro expands to reference
private internal functions of the standard library that cannot be called
directly without the `fmt_internals` feature gate.
More history about this may be found in
[#23616](https://github.com/rust-lang/rust/issues/23616#issuecomment-484999901).
* `pp-exact` is used to ensure a pretty-print test results in specific output.
- `pp-exact` is used to ensure a pretty-print test results in specific output.
If specified without a value, then it means the pretty-print output should
match the original source.
If specified with a value, as in `// pp-exact:foo.pp`,
it will ensure that the pretty-printed output matches the contents of the given file.
Otherwise, if `pp-exact` is not specified, then the pretty-printed output
will be pretty-printed one more time, and the output of the two
pretty-printing rounds will be compared to ensure that the pretty-printed
output converges to a steady state.
match the original source. If specified with a value, as in `//@
pp-exact:foo.pp`, it will ensure that the pretty-printed output matches the
contents of the given file. Otherwise, if `pp-exact` is not specified, then
the pretty-printed output will be pretty-printed one more time, and the output
of the two pretty-printing rounds will be compared to ensure that the
pretty-printed output converges to a steady state.
[`tests/pretty`]: https://github.com/rust-lang/rust/tree/master/tests/pretty
### Incremental tests
The tests in [`tests/incremental`] exercise incremental compilation.
They use [revision headers](#revisions) to tell compiletest to run the
compiler in a series of steps.
The tests in [`tests/incremental`] exercise incremental compilation. They use
[`revisions` directive](#revisions) to tell compiletest to run the compiler in a
series of steps.
Compiletest starts with an empty directory with the `-C incremental` flag, and
then runs the compiler for each revision, reusing the incremental results from
previous steps.
The revisions should start with:
* `rpass` — the test should compile and run successfully
* `rfail` — the test should compile successfully, but the executable should fail to run
* `cfail` — the test should fail to compile
To make the revisions unique, you should add a suffix like `rpass1` and `rpass2`.
To make the revisions unique, you should add a suffix like `rpass1` and
`rpass2`.
To simulate changing the source, compiletest also passes a `--cfg` flag with the
current revision name.
To simulate changing the source, compiletest also passes a `--cfg` flag with
the current revision name.
For example, this will run twice, simulating changing a function:
```rust,ignore
@ -174,28 +179,27 @@ fn foo() {
fn main() { foo(); }
```
`cfail` tests support the `forbid-output` header to specify that a certain
substring must not appear anywhere in the compiler output.
This can be useful to ensure certain errors do not appear, but this can be
fragile as error messages change over time, and a test may no longer be
checking the right thing but will still pass.
`cfail` tests support the `forbid-output` directive to specify that a certain
substring must not appear anywhere in the compiler output. This can be useful to
ensure certain errors do not appear, but this can be fragile as error messages
change over time, and a test may no longer be checking the right thing but will
still pass.
`cfail` tests support the `should-ice` header to specify that a test should
cause an Internal Compiler Error (ICE).
This is a highly specialized header to check that the incremental cache
continues to work after an ICE.
`cfail` tests support the `should-ice` directive to specify that a test should
cause an Internal Compiler Error (ICE). This is a highly specialized directive
to check that the incremental cache continues to work after an ICE.
[`tests/incremental`]: https://github.com/rust-lang/rust/tree/master/tests/incremental
### Debuginfo tests
The tests in [`tests/debuginfo`] test debuginfo generation.
They build a program, launch a debugger, and issue commands to the debugger.
A single test can work with cdb, gdb, and lldb.
The tests in [`tests/debuginfo`] test debuginfo generation. They build a
program, launch a debugger, and issue commands to the debugger. A single test
can work with cdb, gdb, and lldb.
Most tests should have the `// compile-flags: -g` header or something similar
to generate the appropriate debuginfo.
Most tests should have the `//@ compile-flags: -g` directive or something
similar to generate the appropriate debuginfo.
To set a breakpoint on a line, add a `// #break` comment on the line.
@ -205,15 +209,16 @@ The debuginfo tests consist of a series of debugger commands along with
The commands are comments of the form `// $DEBUGGER-command:$COMMAND` where
`$DEBUGGER` is the debugger being used and `$COMMAND` is the debugger command
to execute.
The debugger values can be:
* `cdb`
* `gdb`
* `gdbg` — GDB without Rust support (versions older than 7.11)
* `gdbr` — GDB with Rust support
* `lldb`
* `lldbg` — LLDB without Rust support
* `lldbr` — LLDB with Rust support (this no longer exists)
- `cdb`
- `gdb`
- `gdbg` — GDB without Rust support (versions older than 7.11)
- `gdbr` — GDB with Rust support
- `lldb`
- `lldbg` — LLDB without Rust support
- `lldbr` — LLDB with Rust support (this no longer exists)
The command to check the output are of the form `// $DEBUGGER-check:$OUTPUT`
where `$OUTPUT` is the output to expect.
@ -237,30 +242,32 @@ fn main() {
fn b() {}
```
The following [header commands](headers.md) are available to disable a
test based on the debugger currently being used:
The following [directives](directives.md) are available to disable a test based on
the debugger currently being used:
* `min-cdb-version: 10.0.18317.1001` — ignores the test if the version of cdb
- `min-cdb-version: 10.0.18317.1001` — ignores the test if the version of cdb
is below the given version
* `min-gdb-version: 8.2` — ignores the test if the version of gdb is below the
- `min-gdb-version: 8.2` — ignores the test if the version of gdb is below the
given version
* `ignore-gdb-version: 9.2` — ignores the test if the version of gdb is equal
- `ignore-gdb-version: 9.2` — ignores the test if the version of gdb is equal
to the given version
* `ignore-gdb-version: 7.11.90 - 8.0.9` — ignores the test if the version of
- `ignore-gdb-version: 7.11.90 - 8.0.9` — ignores the test if the version of
gdb is in a range (inclusive)
* `min-lldb-version: 310` — ignores the test if the version of lldb is below
- `min-lldb-version: 310` — ignores the test if the version of lldb is below
the given version
* `rust-lldb` — ignores the test if lldb is not contain the Rust plugin.
NOTE: The "Rust" version of LLDB doesn't exist anymore, so this will always be ignored.
This should probably be removed.
- `rust-lldb` — ignores the test if lldb is not contain the Rust plugin. NOTE:
The "Rust" version of LLDB doesn't exist anymore, so this will always be
ignored. This should probably be removed.
> **Note on running lldb debuginfo tests locally**
>
> If you want to run lldb debuginfo tests locally, then currently on Windows it is required that:
> If you want to run lldb debuginfo tests locally, then currently on Windows it
> is required that:
>
> - You have Python 3.10 installed.
> - You have `python310.dll` available in your `PATH` env var. This is not provided by the standard
> Python installer you obtain from `python.org`; you need to add this to `PATH` manually.
> - You have `python310.dll` available in your `PATH` env var. This is not
> provided by the standard Python installer you obtain from `python.org`; you
> need to add this to `PATH` manually.
>
> Otherwise the lldb debuginfo tests can produce crashes in mysterious ways.
@ -269,11 +276,11 @@ test based on the debugger currently being used:
### Codegen tests
The tests in [`tests/codegen`] test LLVM code generation.
They compile the test with the `--emit=llvm-ir` flag to emit LLVM IR.
They then run the LLVM [FileCheck] tool.
The test is annotated with various `// CHECK` comments to check the generated code.
See the FileCheck documentation for a tutorial and more information.
The tests in [`tests/codegen`] test LLVM code generation. They compile the test
with the `--emit=llvm-ir` flag to emit LLVM IR. They then run the LLVM
[FileCheck] tool. The test is annotated with various `// CHECK` comments to
check the generated code. See the [FileCheck] documentation for a tutorial and
more information.
See also the [assembly tests](#assembly-tests) for a similar set of tests.
@ -283,18 +290,17 @@ See also the [assembly tests](#assembly-tests) for a similar set of tests.
### Assembly tests
The tests in [`tests/assembly`] test LLVM assembly output.
They compile the test with the `--emit=asm` flag to emit a `.s` file with the
assembly output.
They then run the LLVM [FileCheck] tool.
The tests in [`tests/assembly`] test LLVM assembly output. They compile the test
with the `--emit=asm` flag to emit a `.s` file with the assembly output. They
then run the LLVM [FileCheck] tool.
Each test should be annotated with the `// assembly-output:` header
with a value of either `emit-asm` or `ptx-linker` to indicate
the type of assembly output.
Each test should be annotated with the `//@ assembly-output:` directive with a
value of either `emit-asm` or `ptx-linker` to indicate the type of assembly
output.
Then, they should be annotated with various `// CHECK` comments to check the
assembly output.
See the FileCheck documentation for a tutorial and more information.
assembly output. See the [FileCheck] documentation for a tutorial and more
information.
See also the [codegen tests](#codegen-tests) for a similar set of tests.
@ -310,26 +316,27 @@ These tests work by running `rustc` with a flag to print the result of the
monomorphization collection pass, and then special annotations in the file are
used to compare against that.
Each test should be annotated with the `// compile-flags:-Zprint-mono-items=VAL`
header with the appropriate VAL to instruct `rustc` to print the
monomorphization information.
Each test should be annotated with the `//@
compile-flags:-Zprint-mono-items=VAL` directive with the appropriate `VAL` to
instruct `rustc` to print the monomorphization information.
Then, the test should be annotated with comments of the form `//~ MONO_ITEM name`
where `name` is the monomorphized string printed by rustc like `fn <u32 as Trait>::foo`.
Then, the test should be annotated with comments of the form `//~ MONO_ITEM
name` where `name` is the monomorphized string printed by rustc like `fn <u32 as
Trait>::foo`.
To check for CGU partitioning, a comment of the form `//~ MONO_ITEM name @@ cgu`
where `cgu` is a space separated list of the CGU names and the linkage
information in brackets.
For example: `//~ MONO_ITEM static function::FOO @@ statics[Internal]`
information in brackets. For example: `//~ MONO_ITEM static function::FOO @@
statics[Internal]`
[`tests/codegen-units`]: https://github.com/rust-lang/rust/tree/master/tests/codegen-units
### Mir-opt tests
The tests in [`tests/mir-opt`] check parts of the generated MIR to make
sure it is generated correctly and is doing the expected optimizations.
Check out the [MIR Optimizations](../mir/optimizations.md) chapter for more.
The tests in [`tests/mir-opt`] check parts of the generated MIR to make sure it
is generated correctly and is doing the expected optimizations. Check out the
[MIR Optimizations](../mir/optimizations.md) chapter for more.
Compiletest will build the test with several flags to dump the MIR output and
set a baseline for optimizations:
@ -341,29 +348,28 @@ set a baseline for optimizations:
* `-Zdump-mir-exclude-pass-number`
The test should be annotated with `// EMIT_MIR` comments that specify files that
will contain the expected MIR output.
You can use `x test --bless` to create the initial expected files.
will contain the expected MIR output. You can use `x test --bless` to create the
initial expected files.
There are several forms the `EMIT_MIR` comment can take:
* `// EMIT_MIR $MIR_PATH.mir` — This will check that the given filename
matches the exact output from the MIR dump.
For example, `my_test.main.SimplifyCfg-elaborate-drops.after.mir` will load
that file from the test directory, and compare it against the dump from
rustc.
- `// EMIT_MIR $MIR_PATH.mir` — This will check that the given filename matches
the exact output from the MIR dump. For example,
`my_test.main.SimplifyCfg-elaborate-drops.after.mir` will load that file from
the test directory, and compare it against the dump from rustc.
Checking the "after" file (which is after optimization) is useful if you are
interested in the final state after an optimization.
Some rare cases may want to use the "before" file for completeness.
interested in the final state after an optimization. Some rare cases may want
to use the "before" file for completeness.
* `// EMIT_MIR $MIR_PATH.diff` — where `$MIR_PATH` is the filename of the MIR
dump, such as `my_test_name.my_function.EarlyOtherwiseBranch`.
Compiletest will diff the `.before.mir` and `.after.mir` files, and compare
the diff output to the expected `.diff` file from the `EMIT_MIR` comment.
- `// EMIT_MIR $MIR_PATH.diff` — where `$MIR_PATH` is the filename of the MIR
dump, such as `my_test_name.my_function.EarlyOtherwiseBranch`. Compiletest
will diff the `.before.mir` and `.after.mir` files, and compare the diff
output to the expected `.diff` file from the `EMIT_MIR` comment.
This is useful if you want to see how an optimization changes the MIR.
* `// EMIT_MIR $MIR_PATH.dot` — When using specific flags that dump additional
- `// EMIT_MIR $MIR_PATH.dot` — When using specific flags that dump additional
MIR data (e.g. `-Z dump-mir-graphviz` to produce `.dot` files), this will
check that the output matches the given file.
@ -377,35 +383,33 @@ your test, causing separate files to be generated for 32bit and 64bit systems.
### `run-make` tests
> NOTE:
> **Note on phasing out `Makefile`s**
>
> We are planning to migrate all existing Makefile-based `run-make` tests
> to Rust recipes. You should not be adding new Makefile-based `run-make`
> to Rust programs. You should not be adding new Makefile-based `run-make`
> tests.
>
> See <https://github.com/rust-lang/rust/issues/121876>.
The tests in [`tests/run-make`] are general-purpose tests using Rust *recipes*,
which are small programs allowing arbitrary Rust code such as `rustc`
invocations, and is supported by a [`run_make_support`] library. Using Rust
recipes provide the ultimate in flexibility.
which are small programs (`rmake.rs`) allowing arbitrary Rust code such as
`rustc` invocations, and is supported by a [`run_make_support`] library. Using
Rust recipes provide the ultimate in flexibility.
*These should be used as a last resort*. If possible, you should use one of the
other test suites.
If there is some minor feature missing which you need for your test,
consider extending compiletest to add a header command for what you need.
However, if running a bunch of commands is really what you need,
`run-make` is here to the rescue!
`run-make` tests should be used if no other test suites better suit your needs.
#### Using Rust recipes
Each test should be in a separate directory with a `rmake.rs` Rust program,
called the *recipe*. A recipe will be compiled and executed by compiletest
with the `run_make_support` library linked in.
called the *recipe*. A recipe will be compiled and executed by compiletest with
the `run_make_support` library linked in.
If you need new utilities or functionality, consider extending and improving
the [`run_make_support`] library.
If you need new utilities or functionality, consider extending and improving the
[`run_make_support`] library.
Compiletest directives like `//@ only-<target>` or `//@ ignore-<target>` are supported in
`rmake.rs`, like in UI tests.
Compiletest directives like `//@ only-<target>` or `//@ ignore-<target>` are
supported in `rmake.rs`, like in UI tests. However, revisions or building
auxiliary via directives are not currently supported.
Two `run-make` tests are ported over to Rust recipes as examples:
@ -426,14 +430,16 @@ Of course, some tests will not successfully *run* in this way.
#### Using Makefiles (legacy)
> NOTE:
> You should avoid writing new Makefile-based `run-make` tests.
<div class="warning">
You should avoid writing new Makefile-based `run-make` tests.
</div>
Each test should be in a separate directory with a `Makefile` indicating the
commands to run.
There is a [`tools.mk`] Makefile which you can include which provides a bunch of
utilities to make it easier to run commands and compare outputs.
Take a look at some of the other tests for some examples on how to get started.
utilities to make it easier to run commands and compare outputs. Take a look at
some of the other tests for some examples on how to get started.
[`tools.mk`]: https://github.com/rust-lang/rust/blob/master/tests/run-make/tools.mk
[`tests/run-make`]: https://github.com/rust-lang/rust/tree/master/tests/run-make
@ -442,9 +448,13 @@ Take a look at some of the other tests for some examples on how to get started.
### Valgrind tests
The tests in [`tests/run-pass-valgrind`] are for use with [Valgrind].
These are currently vestigial, as Valgrind is no longer used in CI.
These may be removed in the future.
> **TODO**
>
> Yeet this if we yeet the test suite.
The tests in [`tests/run-pass-valgrind`] are for use with [Valgrind]. These are
currently vestigial, as Valgrind is no longer used in CI. These may be removed
in the future.
[Valgrind]: https://valgrind.org/
[`tests/run-pass-valgrind`]: https://github.com/rust-lang/rust/tree/master/tests/run-pass-valgrind
@ -453,9 +463,8 @@ These may be removed in the future.
### Coverage tests
The tests in [`tests/coverage`] are shared by multiple test modes that test
coverage instrumentation in different ways.
Running the `coverage` test suite will automatically run each test in all of
the different coverage modes.
coverage instrumentation in different ways. Running the `coverage` test suite
will automatically run each test in all of the different coverage modes.
Each mode also has an alias to run the coverage tests in just that mode:
@ -471,35 +480,34 @@ Each mode also has an alias to run the coverage tests in just that mode:
./x test coverage-map -- tests/coverage/if.rs # runs the specified test in "coverage-map" mode only
```
---
#### `coverage-map` suite
In `coverage-map` mode, these tests verify the mappings between source code
regions and coverage counters that are emitted by LLVM.
They compile the test with `--emit=llvm-ir`,
then use a custom tool ([`src/tools/coverage-dump`])
to extract and pretty-print the coverage mappings embedded in the IR.
These tests don't require the profiler runtime, so they run in PR CI jobs and
are easy to run/bless locally.
regions and coverage counters that are emitted by LLVM. They compile the test
with `--emit=llvm-ir`, then use a custom tool ([`src/tools/coverage-dump`]) to
extract and pretty-print the coverage mappings embedded in the IR. These tests
don't require the profiler runtime, so they run in PR CI jobs and are easy to
run/bless locally.
These coverage map tests can be sensitive to changes in MIR lowering or MIR
optimizations, producing mappings that are different but produce identical
coverage reports.
As a rule of thumb, any PR that doesn't change coverage-specific
code should **feel free to re-bless** the `coverage-map` tests as necessary,
without worrying about the actual changes, as long as the `coverage-run` tests
still pass.
As a rule of thumb, any PR that doesn't change coverage-specific code should
**feel free to re-bless** the `coverage-map` tests as necessary, without
worrying about the actual changes, as long as the `coverage-run` tests still
pass.
---
#### `coverage-run` suite
In `coverage-run` mode, these tests perform an end-to-end test of coverage reporting.
They compile a test program with coverage instrumentation, run that program to
produce raw coverage data, and then use LLVM tools to process that data into a
human-readable code coverage report.
In `coverage-run` mode, these tests perform an end-to-end test of coverage
reporting. They compile a test program with coverage instrumentation, run that
program to produce raw coverage data, and then use LLVM tools to process that
data into a human-readable code coverage report.
Instrumented binaries need to be linked against the LLVM profiler runtime,
so `coverage-run` tests are **automatically skipped**
unless the profiler runtime is enabled in `config.toml`:
Instrumented binaries need to be linked against the LLVM profiler runtime, so
`coverage-run` tests are **automatically skipped** unless the profiler runtime
is enabled in `config.toml`:
```toml
# config.toml
@ -507,10 +515,10 @@ unless the profiler runtime is enabled in `config.toml`:
profiler = true
```
This also means that they typically don't run in PR CI jobs,
though they do run as part of the full set of CI jobs used for merging.
This also means that they typically don't run in PR CI jobs, though they do run
as part of the full set of CI jobs used for merging.
---
#### `coverage-run-rustdoc` suite
The tests in [`tests/coverage-run-rustdoc`] also run instrumented doctests and
include them in the coverage report. This avoids having to build rustdoc when
@ -522,27 +530,30 @@ only running the main `coverage` suite.
### Crashes tests
[`tests/crashes`] serve as a collection of tests that are expected to cause the compiler to ICE, panic
or crash in some other way, so that accidental fixes are tracked. This was formally done at
<https://github.com/rust-lang/glacier> but doing it inside the rust-lang/rust testsuite is more
convenient.
[`tests/crashes`] serve as a collection of tests that are expected to cause the
compiler to ICE, panic or crash in some other way, so that accidental fixes are
tracked. This was formally done at <https://github.com/rust-lang/glacier> but
doing it inside the rust-lang/rust testsuite is more convenient.
It is imperative that a test in the suite causes rustc to ICE, panic or crash crash in some other
way. A test will "pass" if rustc exits with an exit status other than 1 or 0.
It is imperative that a test in the suite causes rustc to ICE, panic or crash
crash in some other way. A test will "pass" if rustc exits with an exit status
other than 1 or 0.
If you want to see verbose stdout/stderr, you need to set `COMPILETEST_VERBOSE_CRASHES=1`, e.g.
If you want to see verbose stdout/stderr, you need to set
`COMPILETEST_VERBOSE_CRASHES=1`, e.g.
```bash
$ COMPILETEST_VERBOSE_CRASHES=1 ./x test tests/crashes/999999.rs --stage 1
```
When adding crashes from <https://github.com/rust-lang/rust/issues>, the issue number should be
noted in the file name (`12345.rs` should suffice) and also inside the file include a `//@ known-bug
#4321` directive.
When adding crashes from <https://github.com/rust-lang/rust/issues>, the issue
number should be noted in the file name (`12345.rs` should suffice) and also
inside the file include a `//@ known-bug: #4321` directive.
If you happen to fix one of the crashes, please move it to a fitting subdirectory in `tests/ui` and
give it a meaningful name. Please add a doc comment at the top of the file explaining why this test
exists, even better if you can briefly explain how the example causes rustc to crash previously and
If you happen to fix one of the crashes, please move it to a fitting
subdirectory in `tests/ui` and give it a meaningful name. Please add a doc
comment at the top of the file explaining why this test exists, even better if
you can briefly explain how the example causes rustc to crash previously and
what was done to prevent rustc to ICE/panic/crash.
Adding
@ -554,24 +565,25 @@ Fixes #MMMMM
to the description of your pull request will ensure the corresponding tickets be closed
automatically upon merge.
Make sure that your fix actually fixes the root cause of the issue and not just a subset first.
The issue numbers can be found in the file name or the `//@ known-bug`
directive inside the test file.
Make sure that your fix actually fixes the root cause of the issue and not just
a subset first. The issue numbers can be found in the file name or the `//@
known-bug` directive inside the test file.
[`tests/crashes`]: https://github.com/rust-lang/rust/tree/master/tests/crashes
## Building auxiliary crates
It is common that some tests require additional auxiliary crates to be compiled.
There are multiple [headers](headers.md) to assist with that:
There are multiple [directives](directives.md) to assist with that:
* `aux-build`
* `aux-crate`
* `aux-bin`
* `aux-codegen-backend`
- `aux-build`
- `aux-crate`
- `aux-bin`
- `aux-codegen-backend`
`aux-build` will build a separate crate from the named source file.
The source file should be in a directory called `auxiliary` beside the test file.
`aux-build` will build a separate crate from the named source file. The source
file should be in a directory called `auxiliary` beside the test file.
```rust,ignore
//@ aux-build: my-helper.rs
@ -581,16 +593,14 @@ extern crate my_helper;
```
The aux crate will be built as a dylib if possible (unless on a platform that
does not support them, or the `no-prefer-dynamic` header is specified in the
aux file).
The `-L` flag is used to find the extern crates.
does not support them, or the `no-prefer-dynamic` header is specified in the aux
file). The `-L` flag is used to find the extern crates.
`aux-crate` is very similar to `aux-build`; however, it uses the `--extern`
flag to link to the extern crate.
That allows you to specify the additional syntax of the `--extern` flag, such
as renaming a dependency.
For example, `// aux-crate:foo=bar.rs` will compile `auxiliary/bar.rs` and
make it available under then name `foo` within the test.
`aux-crate` is very similar to `aux-build`. However, it uses the `--extern` flag
to link to the extern crate to make the crate be available as an extern prelude.
That allows you to specify the additional syntax of the `--extern` flag, such as
renaming a dependency. For example, `// aux-crate:foo=bar.rs` will compile
`auxiliary/bar.rs` and make it available under then name `foo` within the test.
This is similar to how Cargo does dependency renaming.
`aux-bin` is similar to `aux-build` but will build a binary instead of a
@ -605,8 +615,9 @@ for tests in `tests/ui-fulldeps`, since it requires the use of compiler crates.
If you want a proc-macro dependency, then there currently is some ceremony
needed.
Place the proc-macro itself in a file like `auxiliary/my-proc-macro.rs`
with the following structure:
Place the proc-macro itself in a file like `auxiliary/my-proc-macro.rs` with the
following structure:
```rust,ignore
//@ force-host
@ -623,10 +634,10 @@ pub fn foo(input: TokenStream) -> TokenStream {
}
```
The `force-host` is needed because proc-macros are loaded in the host
compiler, and `no-prefer-dynamic` is needed to tell compiletest to not use
`prefer-dynamic` which is not compatible with proc-macros.
The `#![crate_type]` attribute is needed to specify the correct crate-type.
The `force-host` is needed because proc-macros are loaded in the host compiler,
and `no-prefer-dynamic` is needed to tell compiletest to not use
`prefer-dynamic` which is not compatible with proc-macros. The `#![crate_type]`
attribute is needed to specify the correct crate-type.
Then in your test, you can build with `aux-build`:
@ -643,21 +654,20 @@ fn main() {
## Revisions
Revisions allow a single test file to be used for multiple tests.
This is done by adding a special header at the top of the file:
Revisions allow a single test file to be used for multiple tests. This is done
by adding a special directive at the top of the file:
```rust,ignore
//@ revisions: foo bar baz
```
This will result in the test being compiled (and tested) three times,
once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg
baz`.
You can therefore use `#[cfg(foo)]` etc within the test to tweak
each of these results.
This will result in the test being compiled (and tested) three times, once with
`--cfg foo`, once with `--cfg bar`, and once with `--cfg baz`. You can therefore
use `#[cfg(foo)]` etc within the test to tweak each of these results.
You can also customize headers and expected error messages to a particular
revision. To do this, add `[revision-name]` after the `//` comment, like so:
You can also customize directives and expected error messages to a particular
revision. To do this, add `[revision-name]` after the `//@` for directives, and
after `//` for UI error annotations, like so:
```rust,ignore
// A flag to pass in only for cfg `foo`:
@ -677,8 +687,8 @@ also registered as an additional prefix for FileCheck directives:
```rust,ignore
//@ revisions: NORMAL COVERAGE
//@ [COVERAGE] compile-flags: -Cinstrument-coverage
//@ [COVERAGE] needs-profiler-support
//@[COVERAGE] compile-flags: -Cinstrument-coverage
//@[COVERAGE] needs-profiler-support
// COVERAGE: @__llvm_coverage_mapping
// NORMAL-NOT: @__llvm_coverage_mapping
@ -687,45 +697,46 @@ also registered as an additional prefix for FileCheck directives:
fn main() {}
```
Note that not all headers have meaning when customized to a revision.
For example, the `ignore-test` header (and all "ignore" headers)
currently only apply to the test as a whole, not to particular
revisions. The only headers that are intended to really work when
customized to a revision are error patterns and compiler flags.
Note that not all directives have meaning when customized to a revision. For
example, the `ignore-test` directives (and all "ignore" directives) currently
only apply to the test as a whole, not to particular revisions. The only
directives that are intended to really work when customized to a revision are
error patterns and compiler flags.
<!-- date-check jul 2023 -->
Following is classes of tests that support revisions:
- UI
The following test suites support revisions:
- ui
- assembly
- codegen
- coverage
- debuginfo
- rustdoc UI tests
- incremental (these are special in that they inherently cannot be run in parallel)
- incremental (these are special in that they inherently cannot be run in
parallel)
### Ignoring unused revision names
Normally, revision names mentioned in other headers and error annotations must
correspond to an actual revision declared in a `revisions` header. This is
Normally, revision names mentioned in other directives and error annotations
must correspond to an actual revision declared in a `revisions` directive. This is
enforced by an `./x test tidy` check.
If a revision name needs to be temporarily removed from the revision list for
some reason, the above check can be suppressed by adding the revision name to
an `//@ unused-revision-names:` header instead.
some reason, the above check can be suppressed by adding the revision name to an
`//@ unused-revision-names:` header instead.
Specifying an unused name of `*` (i.e. `//@ unused-revision-names: *`) will
permit any unused revision name to be mentioned.
## Compare modes
Compiletest can be run in different modes, called _compare modes_, which can
be used to compare the behavior of all tests with different compiler flags
enabled.
Compiletest can be run in different modes, called _compare modes_, which can be
used to compare the behavior of all tests with different compiler flags enabled.
This can help highlight what differences might appear with certain flags, and
check for any problems that might arise.
To run the tests in a different mode, you need to pass the `--compare-mode`
CLI flag:
To run the tests in a different mode, you need to pass the `--compare-mode` CLI
flag:
```bash
./x test tests/ui --compare-mode=chalk
@ -733,10 +744,12 @@ CLI flag:
The possible compare modes are:
* `polonius` — Runs with Polonius with `-Zpolonius`.
* `chalk` — Runs with Chalk with `-Zchalk`.
* `split-dwarf` — Runs with unpacked split-DWARF with `-Csplit-debuginfo=unpacked`.
* `split-dwarf-single` — Runs with packed split-DWARF with `-Csplit-debuginfo=packed`.
- `polonius` — Runs with Polonius with `-Zpolonius`.
- `chalk` — Runs with Chalk with `-Zchalk`.
- `split-dwarf` — Runs with unpacked split-DWARF with
`-Csplit-debuginfo=unpacked`.
- `split-dwarf-single` — Runs with packed split-DWARF with
`-Csplit-debuginfo=packed`.
See [UI compare modes](ui.md#compare-modes) for more information about how UI
tests support different output for different modes.
@ -744,10 +757,9 @@ tests support different output for different modes.
In CI, compare modes are only used in one Linux builder, and only with the
following settings:
* `tests/debuginfo`: Uses `split-dwarf` mode.
This helps ensure that none of the debuginfo tests are affected when
enabling split-DWARF.
- `tests/debuginfo`: Uses `split-dwarf` mode. This helps ensure that none of the
debuginfo tests are affected when enabling split-DWARF.
Note that compare modes are separate to [revisions](#revisions).
All revisions are tested when running `./x test tests/ui`, however
compare-modes must be manually run individually via the `--compare-mode` flag.
Note that compare modes are separate to [revisions](#revisions). All revisions
are tested when running `./x test tests/ui`, however compare-modes must be
manually run individually via the `--compare-mode` flag.

30
src/tests/crater.md
View File

@ -1,10 +1,10 @@
# Crater
[Crater](https://github.com/rust-lang/crater) is a tool for compiling
and running tests for _every_ crate on [crates.io](https://crates.io) (and a
few on GitHub). It is mainly used for checking the extent of breakage when
implementing potentially breaking changes and ensuring lack of breakage by
running beta vs stable compiler versions.
[Crater](https://github.com/rust-lang/crater) is a tool for compiling and
running tests for _every_ crate on [crates.io](https://crates.io) (and a few on
GitHub). It is mainly used for checking the extent of breakage when implementing
potentially breaking changes and ensuring lack of breakage by running beta vs
stable compiler versions.
## When to run Crater
@ -15,16 +15,16 @@ or could cause breakage. If you are unsure, feel free to ask your PR's reviewer.
The rust team maintains a few machines that can be used for running crater runs
on the changes introduced by a PR. If your PR needs a crater run, leave a
comment for the triage team in the PR thread. Please inform the team whether
you require a "check-only" crater run, a "build only" crater run, or a
comment for the triage team in the PR thread. Please inform the team whether you
require a "check-only" crater run, a "build only" crater run, or a
"build-and-test" crater run. The difference is primarily in time; the
conservative (if you're not sure) option is to go for the build-and-test run.
If making changes that will only have an effect at compile-time (e.g.,
implementing a new trait) then you only need a check run.
conservative (if you're not sure) option is to go for the build-and-test run. If
making changes that will only have an effect at compile-time (e.g., implementing
a new trait) then you only need a check run.
Your PR will be enqueued by the triage team and the results will be posted when
they are ready. Check runs will take around ~3-4 days, with the other two
taking 5-6 days on average.
they are ready. Check runs will take around ~3-4 days, with the other two taking
5-6 days on average.
While crater is really useful, it is also important to be aware of a few
caveats:
@ -37,9 +37,9 @@ caveats:
- Crater only runs Linux builds on x86_64. Thus, other architectures and
platforms are not tested. Critically, this includes Windows.
- Many crates are not tested. This could be for a lot of reasons, including
that the crate doesn't compile any more (e.g. used old nightly features),
has broken or flaky tests, requires network access, or other reasons.
- Many crates are not tested. This could be for a lot of reasons, including that
the crate doesn't compile any more (e.g. used old nightly features), has
broken or flaky tests, requires network access, or other reasons.
- Before crater can be run, `@bors try` needs to succeed in building artifacts.
This means that if your code doesn't compile, you cannot run crater.

2
src/tests/headers.md → src/tests/directives.md
View File

@ -2,6 +2,8 @@
<!-- toc -->
> **FIXME(jieyouxu)** completely revise this chapter.
Header commands are special comments that tell compiletest how to build and
interpret a test.
They must appear before the Rust source in the test.

28
src/tests/ecosystem.md Normal file
View File

@ -0,0 +1,28 @@
# Ecosystem testing
Rust tests integration with real-world code in the ecosystem to catch
regressions and make informed decisions about the evolution of the language.
## Testing methods
### Crater
Crater is a tool which runs tests on many thousands of public projects. This
tool has its own separate infrastructure for running, and is not run as part of
CI. See the [Crater chapter](crater.md) for more details.
### `cargotest`
`cargotest` is a small tool which runs `cargo test` on a few sample projects
(such as `servo`, `ripgrep`, `tokei`, etc.). This runs as part of CI and ensures
there aren't any significant regressions.
> Example: `./x test src/tools/cargotest`
### Large OSS Project builders
We have CI jobs that build large open-source Rust projects that are used as
regression tests in CI. Our integration jobs build the following projects:
- [Fuchsia](fuchsia.md)
- [Rust for Linux](rust-for-linux.md)

19
src/tests/fuchsia.md
View File

@ -36,7 +36,7 @@ See the [Testing with Docker](docker.md) chapter for more details on how to run
and debug jobs with Docker.
Note that a Fuchsia checkout is *large* – as of this writing, a checkout and
build takes 46G of space – and as you might imagine, it takes awhile to
build takes 46G of space – and as you might imagine, it takes a while to
complete.
### Modifying the Fuchsia checkout
@ -65,11 +65,12 @@ to add this to your `$PATH` for some workflows.
There are a few `fx` subcommands that are relevant, including:
* `fx set` accepts build arguments, writes them to `out/default/args.gn`, and runs GN.
* `fx build` builds the Fuchsia project using Ninja. It will automatically pick
- `fx set` accepts build arguments, writes them to `out/default/args.gn`, and
runs GN.
- `fx build` builds the Fuchsia project using Ninja. It will automatically pick
up changes to build arguments and rerun GN. By default it builds everything,
but it also accepts target paths to build specific targets (see below).
* `fx clippy` runs Clippy on specific Rust targets (or all of them). We use this
- `fx clippy` runs Clippy on specific Rust targets (or all of them). We use this
in the Rust CI build to avoid running codegen on most Rust targets. Underneath
it invokes Ninja, just like `fx build`. The clippy results are saved in json
files inside the build output directory before being printed.
@ -94,8 +95,8 @@ and can also be used in `fx build`.
#### Modifying compiler flags
You can put custom compiler flags inside a GN `config` that is added to a target.
As a simple example:
You can put custom compiler flags inside a GN `config` that is added to a
target. As a simple example:
```
config("everybody_loops") {
@ -162,6 +163,6 @@ rustc book][platform-support].
[`//build/config:compiler`]: https://cs.opensource.google/fuchsia/fuchsia/+/main:build/config/BUILD.gn;l=121;drc=c26c473bef93b33117ae417893118907a026fec7
[build system]: https://fuchsia.dev/fuchsia-src/development/build/build_system
[^loc]: As of June 2024, Fuchsia had about 2 million lines of first-party Rust code
and a roughly equal amount of third-party code, as counted by tokei (excluding
comments and blanks).
[^loc]: As of June 2024, Fuchsia had about 2 million lines of first-party Rust
code and a roughly equal amount of third-party code, as counted by tokei
(excluding comments and blanks).

49
src/tests/integration.md
View File

@ -1,49 +0,0 @@
# Integration testing
Rust tests integration with real-world code to catch regressions and make
informed decisions about the evolution of the language.
## Testing methods
### Crater
Crater is a tool which runs tests on many thousands of public projects. This
tool has its own separate infrastructure for running, and is not run as part of
CI. See the [Crater chapter](crater.md) for more details.
### Cargo test
`cargotest` is a small tool which runs `cargo test` on a few sample projects
(such as `servo`, `ripgrep`, `tokei`, etc.).
This runs as part of CI and ensures there aren't any significant regressions.
> Example: `./x test src/tools/cargotest`
### Integration builders
Integration jobs build large open-source Rust projects that are used as
regression tests in CI. Our integration jobs build the following projects:
- [Fuchsia](fuchsia.md)
- [Rust for Linux](rust-for-linux.md)
## A note about terminology
The term "integration testing" can be used to mean many things. Many of the
compiletest tests within the Rust repo could be justifiably called integration
tests, because they test the integration of many parts of the compiler, or test
the integration of the compiler with other external tools. Calling all of them
integration tests would not be very helpful, especially since those kinds of
tests already have their own specialized names.
We use the term "integration" here to mean integrating the Rust compiler and
toolchain with the ecosystem of Rust projects that depend on it. This is partly
for lack of a better term, but it also reflects a difference in testing approach
from other projects and the comparative advantage it implies.
The Rust compiler is part of the ecosystem, and the ecosystem is in many cases
part of Rust, both in terms of libraries it uses and in terms of the efforts of many
contributors who come to "scratch their own itch". Finally, because Rust has the
ability to do integration testing at such a broad scale, it shortens development
cycles by finding defects earlier.

101
src/tests/intro.md
View File

@ -2,10 +2,10 @@
<!-- toc -->
The Rust project runs a wide variety of different tests, orchestrated by
the build system (`./x test`).
This section gives a brief overview of the different testing tools.
Subsequent chapters dive into [running tests](running.md) and [adding new tests](adding.md).
The Rust project runs a wide variety of different tests, orchestrated by the
build system (`./x test`). This section gives a brief overview of the different
testing tools. Subsequent chapters dive into [running tests](running.md) and
[adding new tests](adding.md).
## Kinds of tests
@ -14,9 +14,13 @@ Almost all of them are driven by `./x test`, with some exceptions noted below.
### Compiletest
The main test harness for testing the compiler itself is a tool called [compiletest].
It supports running different styles of tests, called *test suites*.
The tests are all located in the [`tests`] directory.
The main test harness for testing the compiler itself is a tool called
[compiletest].
[compiletest] supports running different styles of tests, organized into *test
suites*. A *test mode* may provide common presets/behavior for a set of *test
suites*. [compiletest]-supported tests are located in the [`tests`] directory.
The [Compiletest chapter][compiletest] goes into detail on how to use this tool.
> Example: `./x test tests/ui`
@ -26,10 +30,10 @@ The [Compiletest chapter][compiletest] goes into detail on how to use this tool.
### Package tests
The standard library and many of the compiler packages include typical Rust `#[test]`
unit tests, integration tests, and documentation tests.
You can pass a path to `x` to almost any package in the `library` or `compiler` directory,
and `x` will essentially run `cargo test` on that package.
The standard library and many of the compiler packages include typical Rust
`#[test]` unit tests, integration tests, and documentation tests. You can pass a
path to `./x test` for almost any package in the `library/` or `compiler/`
directory, and `x` will essentially run `cargo test` on that package.
Examples:
@ -39,25 +43,25 @@ Examples:
| `./x test library/core` | Runs tests on `core` only |
| `./x test compiler/rustc_data_structures` | Runs tests on `rustc_data_structures` |
The standard library relies very heavily on documentation tests to cover its functionality.
However, unit tests and integration tests can also be used as needed.
Almost all of the compiler packages have doctests disabled.
The standard library relies very heavily on documentation tests to cover its
functionality. However, unit tests and integration tests can also be used as
needed. Almost all of the compiler packages have doctests disabled.
All standard library and compiler unit tests are placed in separate `tests` file
(which is enforced in [tidy][tidy-unit-tests]).
This ensures that when the test file is changed, the crate does not need to be recompiled.
For example:
(which is enforced in [tidy][tidy-unit-tests]). This ensures that when the test
file is changed, the crate does not need to be recompiled. For example:
```rust,ignore
#[cfg(test)]
mod tests;
```
If it wasn't done this way,
and you were working on something like `core`,
that would require recompiling the entire standard library, and the entirety of `rustc`.
If it wasn't done this way, and you were working on something like `core`, that
would require recompiling the entire standard library, and the entirety of
`rustc`.
`./x test` includes some CLI options for controlling the behavior with these tests:
`./x test` includes some CLI options for controlling the behavior with these
package tests:
* `--doc` — Only runs documentation tests in the package.
* `--no-doc` — Run all tests *except* documentation tests.
@ -66,16 +70,18 @@ that would require recompiling the entire standard library, and the entirety of
### Tidy
Tidy is a custom tool used for validating source code style and formatting conventions,
such as rejecting long lines.
There is more information in the [section on coding conventions](../conventions.md#formatting).
Tidy is a custom tool used for validating source code style and formatting
conventions, such as rejecting long lines. There is more information in the
[section on coding conventions](../conventions.md#formatting).
> Examples: `./x test tidy`
> Example: `./x test tidy`
### Formatting
Rustfmt is integrated with the build system to enforce uniform style across the compiler.
The formatting check is automatically run by the Tidy tool mentioned above.
Rustfmt is integrated with the build system to enforce uniform style across the
compiler. The formatting check is automatically run by the Tidy tool mentioned
above.
Examples:
@ -87,10 +93,10 @@ Examples:
### Book documentation tests
All of the books that are published have their own tests,
primarily for validating that the Rust code examples pass.
Under the hood, these are essentially using `rustdoc --test` on the markdown files.
The tests can be run by passing a path to a book to `./x test`.
All of the books that are published have their own tests, primarily for
validating that the Rust code examples pass. Under the hood, these are
essentially using `rustdoc --test` on the markdown files. The tests can be run
by passing a path to a book to `./x test`.
> Example: `./x test src/doc/book`
@ -106,47 +112,48 @@ This requires building all of the documentation, which might take a while.
### Dist check
`distcheck` verifies that the source distribution tarball created by the build system
will unpack, build, and run all tests.
`distcheck` verifies that the source distribution tarball created by the build
system will unpack, build, and run all tests.
> Example: `./x test distcheck`
### Tool tests
Packages that are included with Rust have all of their tests run as well.
This includes things such as cargo, clippy, rustfmt, miri, bootstrap
(testing the Rust build system itself), etc.
Packages that are included with Rust have all of their tests run as well. This
includes things such as cargo, clippy, rustfmt, miri, bootstrap (testing the
Rust build system itself), etc.
Most of the tools are located in the [`src/tools`] directory.
To run the tool's tests, just pass its path to `./x test`.
Most of the tools are located in the [`src/tools`] directory. To run the tool's
tests, just pass its path to `./x test`.
> Example: `./x test src/tools/cargo`
Usually these tools involve running `cargo test` within the tool's directory.
If you want to run only a specified set of tests, append `--test-args FILTER_NAME` to the command.
If you want to run only a specified set of tests, append `--test-args
FILTER_NAME` to the command.
> Example: `./x test src/tools/miri --test-args padding`
In CI, some tools are allowed to fail.
Failures send notifications to the corresponding teams, and is tracked on the [toolstate website].
More information can be found in the [toolstate documentation].
In CI, some tools are allowed to fail. Failures send notifications to the
corresponding teams, and is tracked on the [toolstate website]. More information
can be found in the [toolstate documentation].
[`src/tools`]: https://github.com/rust-lang/rust/tree/master/src/tools/
[toolstate documentation]: https://forge.rust-lang.org/infra/toolstate.html
[toolstate website]: https://rust-lang-nursery.github.io/rust-toolstate/
### Integration testing
### Ecosystem testing
Rust tests integration with real-world code to catch regressions and make
informed decisions about the evolution of the language. There are several kinds
of integration tests, including Crater. See the [Integration testing
chapter](integration.md) for more details.
of ecosystem tests, including Crater. See the [Ecosystem testing
chapter](ecosystem.md) for more details.
### Performance testing
A separate infrastructure is used for testing and tracking performance of the compiler.
See the [Performance testing chapter](perf.md) for more details.
A separate infrastructure is used for testing and tracking performance of the
compiler. See the [Performance testing chapter](perf.md) for more details.
## Further reading

47
src/tests/perf.md
View File

@ -4,14 +4,15 @@
A lot of work is put into improving the performance of the compiler and
preventing performance regressions.
The [rustc-perf](https://github.com/rust-lang/rustc-perf) project provides
several services for testing and tracking performance.
It provides hosted infrastructure for running benchmarks as a service.
At this time, only `x86_64-unknown-linux-gnu` builds are tracked.
several services for testing and tracking performance. It provides hosted
infrastructure for running benchmarks as a service. At this time, only
`x86_64-unknown-linux-gnu` builds are tracked.
A "perf run" is used to compare the performance of the compiler in different
configurations for a large collection of popular crates.
Different configurations include "fresh builds", builds with incremental compilation, etc.
configurations for a large collection of popular crates. Different
configurations include "fresh builds", builds with incremental compilation, etc.
The result of a perf run is a comparison between two versions of the compiler
(by their commit hashes).
@ -24,30 +25,29 @@ Any changes are noted in a comment on the PR.
### Manual perf runs
Additionally, performance tests can be ran before a PR is merged on an as-needed basis.
You should request a perf run if your PR may affect performance, especially if
it can affect performance adversely.
Additionally, performance tests can be ran before a PR is merged on an as-needed
basis. You should request a perf run if your PR may affect performance,
especially if it can affect performance adversely.
To evaluate the performance impact of a PR, write this comment on the PR:
`@bors try @rust-timer queue`
> **Note**: Only users authorized to do perf runs are allowed to post this comment.
> Teams that are allowed to use it are tracked in the [Teams
> repository](https://github.com/rust-lang/team) with the `perf = true` value
> in the `[permissions]` section (and bors permissions are also required).
> If you are not on one of those teams, feel free to ask for someone to post
> it for you (either on Zulip or ask the assigned reviewer).
> **Note**: Only users authorized to do perf runs are allowed to post this
> comment. Teams that are allowed to use it are tracked in the [Teams
> repository](https://github.com/rust-lang/team) with the `perf = true` value in
> the `[permissions]` section (and bors permissions are also required). If you
> are not on one of those teams, feel free to ask for someone to post it for you
> (either on Zulip or ask the assigned reviewer).
This will first tell bors to do a "try" build which do a full release build
for `x86_64-unknown-linux-gnu`.
After the build finishes, it will place it in the queue to run the performance
suite against it.
After the performance tests finish, the bot will post a comment on the PR with
a summary and a link to a full report.
This will first tell bors to do a "try" build which do a full release build for
`x86_64-unknown-linux-gnu`. After the build finishes, it will place it in the
queue to run the performance suite against it. After the performance tests
finish, the bot will post a comment on the PR with a summary and a link to a
full report.
If you want to do a perf run for an already built artifact (e.g. for a previous try
build that wasn't benchmarked yet), you can run this instead:
If you want to do a perf run for an already built artifact (e.g. for a previous
try build that wasn't benchmarked yet), you can run this instead:
`@rust-timer build <commit-sha>`
@ -56,5 +56,6 @@ You cannot benchmark the same artifact twice though.
More information about the available perf bot commands can be found
[here](https://perf.rust-lang.org/help.html).
More details about the benchmarking process itself are available in the [perf collector
More details about the benchmarking process itself are available in the [perf
collector
documentation](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md).

245
src/tests/running.md
View File

@ -2,68 +2,84 @@
<!-- toc -->
You can run the tests using `x`. The most basic command – which
you will almost never want to use! – is as follows:
You can run the entire test collection using `x`. But note that running the
*entire* test collection is almost never what you want to do during local
development because it takes a really long time. For local development, see the
subsection after on how to run a subset of tests.
<div class="warning">
Running plain `./x test` will build the stage 1 compiler and then run the whole
test suite. This not only include `tests/`, but also `library/`, `compiler/`,
`src/tools/` package tests and more.
You usually only want to run a subset of the test suites (or even a smaller set
of tests than that) which you expect will exercise your changes. PR CI exercises
a subset of test collections, and merge queue CI will exercise all of the test
collection.
</div>
```bash
./x test
```
This will build the stage 1 compiler and then run the whole test
suite. You probably don't want to do this very often, because it takes
a very long time, and anyway bors / GitHub Actions will do it for you.
(Often, I will run this command in the background after opening a PR that
I think is done, but rarely otherwise. -nmatsakis)
The test results are cached and previously successful tests are `ignored` during
testing. The stdout/stderr contents as well as a timestamp file for every test
can be found under `build/<target-triple>/test/` for the given
`<target-triple>`. To force-rerun a test (e.g. in case the test runner fails to
notice a change) you can use the `--force-rerun` CLI option.
The test results are cached and previously successful tests are
`ignored` during testing. The stdout/stderr contents as well as a
timestamp file for every test can be found under `build/ARCH/test/`.
To force-rerun a test (e.g. in case the test runner fails to notice a change)
you can simply remove the timestamp file, or use the `--force-rerun` CLI
option.
Note that some tests require a Python-enabled gdb. You can test if
your gdb install supports Python by using the `python` command from
within gdb. Once invoked you can type some Python code (e.g.
`print("hi")`) followed by return and then `CTRL+D` to execute it.
If you are building gdb from source, you will need to configure with
`--with-python=<path-to-python-binary>`.
> **Note on requirements of external dependencies**
>
> Some test suites may require external dependecies. This is especially true of
> debuginfo tests. Some debuginfo tests require a Python-enabled gdb. You can
> test if your gdb install supports Python by using the `python` command from
> within gdb. Once invoked you can type some Python code (e.g. `print("hi")`)
> followed by return and then `CTRL+D` to execute it. If you are building gdb
> from source, you will need to configure with
> `--with-python=<path-to-python-binary>`.
## Running a subset of the test suites
When working on a specific PR, you will usually want to run a smaller
set of tests. For example, a good "smoke test" that can be used after
modifying rustc to see if things are generally working correctly would be the
following:
When working on a specific PR, you will usually want to run a smaller set of
tests. For example, a good "smoke test" that can be used after modifying rustc
to see if things are generally working correctly would be to exercise the `ui`
test suite ([`tests/ui`]):
```bash
./x test tests/ui
```
This will run the `ui` test suite. Of course, the choice
of test suites is somewhat arbitrary, and may not suit the task you are
doing. For example, if you are hacking on debuginfo, you may be better off
with the debuginfo test suite:
This will run the `ui` test suite. Of course, the choice of test suites is
somewhat arbitrary, and may not suit the task you are doing. For example, if you
are hacking on debuginfo, you may be better off with the debuginfo test suite:
```bash
./x test tests/debuginfo
```
If you only need to test a specific subdirectory of tests for any
given test suite, you can pass that directory to `./x test`:
If you only need to test a specific subdirectory of tests for any given test
suite, you can pass that directory as a filter to `./x test`:
```bash
./x test tests/ui/const-generics
```
> **Note for MSYS2**
>
> On MSYS2 the paths seem to be strange and `./x test` neither recognizes
> `tests/ui/const-generics` nor `tests\ui\const-generics`. In that case, you can
> workaround it by using e.g. `./x test ui
> --test-args="tests/ui/const-generics"`.
Likewise, you can test a single file by passing its path:
```bash
./x test tests/ui/const-generics/const-test.rs
```
`x` doesn't support running a single tool test by passing its path yet.
You'll have to use the `--test-args` argument as describled [below](#running-an-individual-test).
`x` doesn't support running a single tool test by passing its path yet. You'll
have to use the `--test-args` argument as describled
[below](#running-an-individual-test).
```bash
./x test src/tools/miri --test-args tests/fail/uninit/padding-enum.rs
@ -81,8 +97,8 @@ You'll have to use the `--test-args` argument as describled [below](#running-an-
./x test --stage 0 library/std
```
Note that this only runs tests on `std`; if you want to test `core` or other crates,
you have to specify those explicitly.
Note that this only runs tests on `std`; if you want to test `core` or other
crates, you have to specify those explicitly.
### Run the tidy script and tests on the standard library
@ -96,19 +112,23 @@ you have to specify those explicitly.
./x test --stage 1 library/std
```
By listing which test suites you want to run you avoid having to run
tests for components you did not change at all.
By listing which test suites you want to run you avoid having to run tests for
components you did not change at all.
**Warning:** Note that bors only runs the tests with the full stage 2
build; therefore, while the tests **usually** work fine with stage 1,
there are some limitations.
<div class="warning">
Note that bors only runs the tests with the full stage 2 build; therefore, while
the tests **usually** work fine with stage 1, there are some limitations.
</div>
### Run all tests using a stage 2 compiler
```bash
./x test --stage 2
```
<div class="warning">
You almost never need to do this; CI will run these tests for you.
</div>
## Run unit tests on the compiler/library
@ -126,19 +146,19 @@ But unfortunately, it's impossible. You should invoke the following instead:
## Running an individual test
Another common thing that people want to do is to run an **individual
test**, often the test they are trying to fix. As mentioned earlier,
you may pass the full file path to achieve this, or alternatively one
may invoke `x` with the `--test-args` option:
Another common thing that people want to do is to run an **individual test**,
often the test they are trying to fix. As mentioned earlier, you may pass the
full file path to achieve this, or alternatively one may invoke `x` with the
`--test-args` option:
```bash
./x test tests/ui --test-args issue-1234
```
Under the hood, the test runner invokes the standard Rust test runner
(the same one you get with `#[test]`), so this command would wind up
filtering for tests that include "issue-1234" in the name. (Thus
`--test-args` is a good way to run a collection of related tests.)
Under the hood, the test runner invokes the standard Rust test runner (the same
one you get with `#[test]`), so this command would wind up filtering for tests
that include "issue-1234" in the name. Thus, `--test-args` is a good way to run
a collection of related tests.
## Passing arguments to `rustc` when running tests
@ -151,9 +171,9 @@ additional arguments to the compiler when building the tests.
## Editing and updating the reference files
If you have changed the compiler's output intentionally, or you are
making a new test, you can pass `--bless` to the test subcommand. E.g.
if some tests in `tests/ui` are failing, you can run
If you have changed the compiler's output intentionally, or you are making a new
test, you can pass `--bless` to the test subcommand. E.g. if some tests in
`tests/ui` are failing, you can run
```text
./x test tests/ui --bless
@ -167,37 +187,35 @@ all tests. Of course you can also target just specific tests with the
There are a few options for running tests:
* `config.toml` has the `rust.verbose-tests` option.
If `false`, each test will print a single dot (the default).
If `true`, the name of every test will be printed.
This is equivalent to the `--quiet` option in the [Rust test
* `config.toml` has the `rust.verbose-tests` option. If `false`, each test will
print a single dot (the default). If `true`, the name of every test will be
printed. This is equivalent to the `--quiet` option in the [Rust test
harness](https://doc.rust-lang.org/rustc/tests/)
* The environment variable `RUST_TEST_THREADS` can be set to the number of
concurrent threads to use for testing.
## Passing `--pass $mode`
Pass UI tests now have three modes, `check-pass`, `build-pass` and
`run-pass`. When `--pass $mode` is passed, these tests will be forced
to run under the given `$mode` unless the directive `// ignore-pass`
exists in the test file. For example, you can run all the tests in
`tests/ui` as `check-pass`:
Pass UI tests now have three modes, `check-pass`, `build-pass` and `run-pass`.
When `--pass $mode` is passed, these tests will be forced to run under the given
`$mode` unless the directive `//@ ignore-pass` exists in the test file. For
example, you can run all the tests in `tests/ui` as `check-pass`:
```bash
./x test tests/ui --pass check
```
By passing `--pass $mode`, you can reduce the testing time. For each
mode, please see [Controlling pass/fail
By passing `--pass $mode`, you can reduce the testing time. For each mode,
please see [Controlling pass/fail
expectations](ui.md#controlling-passfail-expectations).
## Running tests with different "compare modes"
UI tests may have different output depending on certain "modes" that
the compiler is in. For example, when using the Polonius
mode, a test `foo.rs` will first look for expected output in
`foo.polonius.stderr`, falling back to the usual `foo.stderr` if not found.
The following will run the UI test suite in Polonius mode:
UI tests may have different output depending on certain "modes" that the
compiler is in. For example, when using the Polonius mode, a test `foo.rs` will
first look for expected output in `foo.polonius.stderr`, falling back to the
usual `foo.stderr` if not found. The following will run the UI test suite in
Polonius mode:
```bash
./x test tests/ui --compare-mode=polonius
@ -207,25 +225,25 @@ See [Compare modes](compiletest.md#compare-modes) for more details.
## Running tests manually
Sometimes it's easier and faster to just run the test by hand.
Most tests are just `rs` files, so after
[creating a rustup toolchain](../building/how-to-build-and-run.md#creating-a-rustup-toolchain),
you can do something like:
Sometimes it's easier and faster to just run the test by hand. Most tests are
just `.rs` files, so after [creating a rustup
toolchain](../building/how-to-build-and-run.md#creating-a-rustup-toolchain), you
can do something like:
```bash
rustc +stage1 tests/ui/issue-1234.rs
```
This is much faster, but doesn't always work. For example, some tests
include directives that specify specific compiler flags, or which rely
on other crates, and they may not run the same without those options.
This is much faster, but doesn't always work. For example, some tests include
directives that specify specific compiler flags, or which rely on other crates,
and they may not run the same without those options.
## Running `run-make` tests
### Windows
Running the `run-make` test suite on Windows is a bit more involved. There are numerous
prerequisites and environmental requirements:
Running the `run-make` test suite on Windows is a currently bit more involved.
There are numerous prerequisites and environmental requirements:
- Install msys2: <https://www.msys2.org/>
- Specify `MSYS2_PATH_TYPE=inherit` in `msys2.ini` in the msys2 installation directory, run the
@ -236,29 +254,38 @@ prerequisites and environmental requirements:
- `pacman -S binutils`
- `./x test run-make` (`./x test tests/run-make` doesn't work)
There is [on-going work][port-run-make] to not rely on `Makefile`s in the
run-make test suite. Once this work is completed, you can run the entire
`run-make` test suite on native Windows inside `cmd` or `PowerShell` without
needing to install and use MSYS2. As of <!--date-check --> Oct 2024, it is
already possible to run the vast majority of the `run-make` test suite outside
of MSYS2, but there will be failures for the tests that still use `Makefile`s
due to not finding `make`.
## Running tests on a remote machine
Tests may be run on a remote machine (e.g. to test builds for a different
architecture). This is done using `remote-test-client` on the build machine
to send test programs to `remote-test-server` running on the remote machine.
architecture). This is done using `remote-test-client` on the build machine to
send test programs to `remote-test-server` running on the remote machine.
`remote-test-server` executes the test programs and sends the results back to
the build machine. `remote-test-server` provides *unauthenticated remote code
execution* so be careful where it is used.
To do this, first build `remote-test-server` for the remote
machine, e.g. for RISC-V
To do this, first build `remote-test-server` for the remote machine, e.g. for
RISC-V
```sh
./x build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
```
The binary will be created at
`./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy
this over to the remote machine.
`./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy this
over to the remote machine.
On the remote machine, run the `remote-test-server` with the `--bind
0.0.0.0:12345` flag (and optionally `-v` for verbose output). Output should
look like this:
0.0.0.0:12345` flag (and optionally `-v` for verbose output). Output should look
like this:
```sh
$ ./remote-test-server -v --bind 0.0.0.0:12345
starting test server
@ -272,6 +299,7 @@ restrictive IP address when binding.
You can test if the `remote-test-server` is working by connecting to it and
sending `ping\n`. It should reply `pong`:
```sh
$ nc $REMOTE_IP 12345
ping
@ -281,13 +309,15 @@ pong
To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
variable then use `x` as usual. For example, to run `ui` tests for a RISC-V
machine with the IP address `1.2.3.4` use
```sh
export TEST_DEVICE_ADDR="1.2.3.4:12345"
./x test tests/ui --target riscv64gc-unknown-linux-gnu
```
If `remote-test-server` was run with the verbose flag, output on the test machine
may look something like
If `remote-test-server` was run with the verbose flag, output on the test
machine may look something like
```
[...]
run "/tmp/work/test1007/a"
@ -311,31 +341,28 @@ output) may fail without ever running on the remote machine.
## Testing on emulators
Some platforms are tested via an emulator for architectures that aren't
readily available. For architectures where the standard library is well
supported and the host operating system supports TCP/IP networking, see the
above instructions for testing on a remote machine (in this case the
remote machine is emulated).
Some platforms are tested via an emulator for architectures that aren't readily
available. For architectures where the standard library is well supported and
the host operating system supports TCP/IP networking, see the above instructions
for testing on a remote machine (in this case the remote machine is emulated).
There is also a set of tools for orchestrating running the
tests within the emulator. Platforms such as `arm-android` and
`arm-unknown-linux-gnueabihf` are set up to automatically run the tests under
emulation on GitHub Actions. The following will take a look at how a target's tests
are run under emulation.
There is also a set of tools for orchestrating running the tests within the
emulator. Platforms such as `arm-android` and `arm-unknown-linux-gnueabihf` are
set up to automatically run the tests under emulation on GitHub Actions. The
following will take a look at how a target's tests are run under emulation.
The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
architecture. Included in the Rust tree are the tools [remote-test-client]
and [remote-test-server] which are programs for sending test programs and
libraries to the emulator, and running the tests within the emulator, and
reading the results. The Docker image is set up to launch
`remote-test-server` and the build tools use `remote-test-client` to
communicate with the server to coordinate running tests (see
[src/bootstrap/src/core/build_steps/test.rs]).
architecture. Included in the Rust tree are the tools [remote-test-client] and
[remote-test-server] which are programs for sending test programs and libraries
to the emulator, and running the tests within the emulator, and reading the
results. The Docker image is set up to launch `remote-test-server` and the
build tools use `remote-test-client` to communicate with the server to
coordinate running tests (see [src/bootstrap/src/core/build_steps/test.rs]).
> TODO:
> Is there any support for using an iOS emulator?
> **TODO**
>
> It's also unclear to me how the wasm or asm.js tests are run.
> - Is there any support for using an iOS emulator?
> - It's also unclear to me how the wasm or asm.js tests are run.
[armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/host-x86_64/armhf-gnu/Dockerfile
[QEMU]: https://www.qemu.org/
@ -374,5 +401,9 @@ need to pass the library file path with `LIBRARY_PATH`:
$ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/12/ ./x test compiler/rustc_codegen_gcc/
```
If you encounter bugs or problems, don't hesitate to open issues on
[rustc_codegen_gcc repository](https://github.com/rust-lang/rustc_codegen_gcc/).
If you encounter bugs or problems, don't hesitate to open issues on the
[`rustc_codegen_gcc`
repository](https://github.com/rust-lang/rustc_codegen_gcc/).
[`tests/ui`]: https://github.com/rust-lang/rust/tree/master/tests/ui
[port-run-make]: https://github.com/rust-lang/rust/issues/121876

39
src/tests/rust-for-linux.md
View File

@ -1,32 +1,45 @@
# Rust for Linux integration tests
[Rust for Linux](https://rust-for-linux.com/) (RfL) is an effort for adding support for the Rust programming
language into the Linux kernel.
[Rust for Linux](https://rust-for-linux.com/) (RfL) is an effort for adding
support for the Rust programming language into the Linux kernel.
## Building Rust for Linux in CI
Rust for Linux builds as part of the suite of bors tests that run before a pull request
is merged.
Rust for Linux builds as part of the suite of bors tests that run before a pull
request is merged.
The workflow builds a stage1 sysroot of the Rust compiler, downloads the Linux kernel, and tries to compile several Rust for Linux drivers and examples using this sysroot. RfL uses several unstable compiler/language features, therefore this workflow notifies us if a given compiler change would break it.
The workflow builds a stage1 sysroot of the Rust compiler, downloads the Linux
kernel, and tries to compile several Rust for Linux drivers and examples using
this sysroot. RfL uses several unstable compiler/language features, therefore
this workflow notifies us if a given compiler change would break it.
If you are worried that a pull request might break the Rust for Linux builder and want
to test it out before submitting it to the bors queue, simply add this line to
your PR description:
If you are worried that a pull request might break the Rust for Linux builder
and want to test it out before submitting it to the bors queue, simply add this
line to your PR description:
> try-job: x86_64-rust-for-linux
Then when you `@bors try` it will pick the job that builds the Rust for Linux integration.
Then when you `@bors try` it will pick the job that builds the Rust for Linux
integration.
## What to do in case of failure
Currently, we use the following unofficial policy for handling failures caused by a change breaking the RfL integration:
Currently, we use the following unofficial policy for handling failures caused
by a change breaking the RfL integration:
- If the breakage was unintentional, then fix the PR.
- If the breakage was intentional, then let [RFL][rfl-ping] know and discuss what will the kernel need to change.
- If the breakage was intentional, then let [RFL][rfl-ping] know and discuss
what will the kernel need to change.
- If the PR is urgent, then disable the test temporarily.
- If the PR can wait a few days, then wait for RFL maintainers to provide a new Linux kernel commit hash with the needed changes done, and apply it to the PR, which would confirm the changes work.
- If the PR can wait a few days, then wait for RFL maintainers to provide a
new Linux kernel commit hash with the needed changes done, and apply it to
the PR, which would confirm the changes work.
If something goes wrong with the workflow, you can ping the [Rust for Linux][rfl-ping] ping group to ask for help.
If something goes wrong with the workflow, you can ping the [Rust for
Linux][rfl-ping] ping group to ask for help.
```text
@rustbot ping rfl
```
[rfl-ping]: ../notification-groups/rust-for-linux.md

42
src/tests/suggest-tests.md
View File

@ -1,36 +1,39 @@
# Suggest tests tool
This chapter is about the internals of and contribution instructions for the
`suggest-tests` tool. For a high-level overview of the tool, see
[this section](../building/suggested.md#x-suggest). This tool is currently in a
beta state and is tracked by [this](https://github.com/rust-lang/rust/issues/109933)
`suggest-tests` tool. For a high-level overview of the tool, see [this
section](../building/suggested.md#x-suggest). This tool is currently in a beta
state and is tracked by [this](https://github.com/rust-lang/rust/issues/109933)
issue on Github. Currently the number of tests it will suggest are very limited
in scope, we are looking to expand this (contributions welcome!).
## Internals
The tool is defined in a separate crate ([`src/tools/suggest-tests`](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests))
The tool is defined in a separate crate
([`src/tools/suggest-tests`](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests))
which outputs suggestions which are parsed by a shim in bootstrap
([`src/bootstrap/src/core/build_steps/suggest.rs`](https://github.com/rust-lang/rust/blob/master/src/bootstrap/src/core/build_steps/suggest.rs)).
The only notable thing the bootstrap shim does is (when invoked with the
`--run` flag) use bootstrap's internal mechanisms to create a new `Builder` and
uses it to invoke the suggested commands. The `suggest-tests` crate is where the
fun happens, two kinds of suggestions are defined: "static" and "dynamic"
The only notable thing the bootstrap shim does is (when invoked with the `--run`
flag) use bootstrap's internal mechanisms to create a new `Builder` and uses it
to invoke the suggested commands. The `suggest-tests` crate is where the fun
happens, two kinds of suggestions are defined: "static" and "dynamic"
suggestions.
### Static suggestions
Defined [here](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/static_suggestions.rs).
Static suggestions are simple: they are just [globs](https://crates.io/crates/glob)
which map to a `x` command. In `suggest-tests`, this is implemented with a
simple `macro_rules` macro.
Defined
[here](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/static_suggestions.rs).
Static suggestions are simple: they are just
[globs](https://crates.io/crates/glob) which map to a `x` command. In
`suggest-tests`, this is implemented with a simple `macro_rules` macro.
### Dynamic suggestions
Defined [here](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/dynamic_suggestions.rs).
Defined
[here](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/dynamic_suggestions.rs).
These are more complicated than static suggestions and are implemented as
functions with the following signature: `fn(&Path) -> Vec<Suggestion>`. In
other words, each suggestion takes a path to a modified file and (after running
functions with the following signature: `fn(&Path) -> Vec<Suggestion>`. In other
words, each suggestion takes a path to a modified file and (after running
arbitrary Rust code) can return any number of suggestions, or none. Dynamic
suggestions are useful for situations where fine-grained control over
suggestions is needed. For example, modifications to the `compiler/xyz/` path
@ -43,13 +46,14 @@ run.
The following steps should serve as a rough guide to add suggestions to
`suggest-tests` (very welcome!):
1. Determine the rules for your suggestion. Is it simple and operates only on
a single path or does it match globs? Does it need fine-grained control over
1. Determine the rules for your suggestion. Is it simple and operates only on a
single path or does it match globs? Does it need fine-grained control over
the resulting command or does "one size fit all"?
2. Based on the previous step, decide if your suggestion should be implemented
as either static or dynamic.
3. Implement the suggestion. If it is dynamic then a test is highly recommended,
to verify that your logic is correct and to give an example of the suggestion.
See the [tests.rs](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/tests.rs)
to verify that your logic is correct and to give an example of the
suggestion. See the
[tests.rs](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/tests.rs)
file.
4. Open a PR implementing your suggestion. **(TODO: add example PR)**

482
src/tests/ui.md
View File

@ -2,105 +2,98 @@
<!-- toc -->
UI tests are a particular [test suite](compiletest.md#test-suites) of compiletest.
UI tests are a particular [test suite](compiletest.md#test-suites) of
compiletest.
## Introduction
The tests in [`tests/ui`] are a collection of general-purpose tests which
primarily focus on validating the console output of the compiler, but can be
used for many other purposes.
For example, tests can also be configured to [run the resulting
program](#controlling-passfail-expectations) to verify its behavior.
used for many other purposes. For example, tests can also be configured to [run
the resulting program](#controlling-passfail-expectations) to verify its
behavior.
[`tests/ui`]: https://github.com/rust-lang/rust/blob/master/tests/ui
## General structure of a test
A test consists of a Rust source file located anywhere in the `tests/ui` directory.
For example, [`tests/ui/hello.rs`] is a basic hello-world test.
A test consists of a Rust source file located anywhere in the `tests/ui`
directory, but they should be placed in a suitable sub-directory. For example,
[`tests/ui/hello.rs`] is a basic hello-world test.
Compiletest will use `rustc` to compile the test, and compare the output
against the expected output which is stored in a `.stdout` or `.stderr` file
located next to the test.
See [Output comparison](#output-comparison) for more.
Compiletest will use `rustc` to compile the test, and compare the output against
the expected output which is stored in a `.stdout` or `.stderr` file located
next to the test. See [Output comparison](#output-comparison) for more.
Additionally, errors and warnings should be annotated with comments within
the source file.
See [Error annotations](#error-annotations) for more.
Additionally, errors and warnings should be annotated with comments within the
source file. See [Error annotations](#error-annotations) for more.
[Headers](headers.md) in the form of comments at the top of the file control
how the test is compiled and what the expected behavior is. Note that tests in
the "ui" test suite require the use of `//@ header-name` instead of
`// header-name` like the other test suites do. The other test suites will be
migrated to use the `//@` syntax too, but that is in progress. Additionally,
`// ignore-tidy` and `// ignore-tidy-*` are ignored by compiletest when
handling "ui" test suite tests (note that they are not `//@` directives).
Compiletest [directives](directives.md) in the form of special comments prefixed
with `//@` control how the test is compiled and what the expected behavior is.
Tests are expected to fail to compile, since most tests are testing compiler
errors.
You can change that behavior with a header, see [Controlling pass/fail
expectations](#controlling-passfail-expectations).
errors. You can change that behavior with a directive, see [Controlling
pass/fail expectations](#controlling-passfail-expectations).
By default, a test is built as an executable binary.
If you need a different crate type, you can use the `#![crate_type]` attribute
to set it as needed.
By default, a test is built as an executable binary. If you need a different
crate type, you can use the `#![crate_type]` attribute to set it as needed.
[`tests/ui/hello.rs`]: https://github.com/rust-lang/rust/blob/master/tests/ui/hello.rs
## Output comparison
UI tests store the expected output from the compiler in `.stderr` and
`.stdout` files next to the test.
You normally generate these files with the `--bless` CLI option, and then
inspect them manually to verify they contain what you expect.
UI tests store the expected output from the compiler in `.stderr` and `.stdout`
snapshots next to the test. You normally generate these files with the `--bless`
CLI option, and then inspect them manually to verify they contain what you
expect.
The output is normalized to ignore unwanted differences, see the
[Normalization](#normalization) section.
If the file is missing, then compiletest expects the corresponding output to
be empty.
[Normalization](#normalization) section. If the file is missing, then
compiletest expects the corresponding output to be empty.
There can be multiple stdout/stderr files.
The general form is:
There can be multiple stdout/stderr files. The general form is:
```text
*test-name*`.`*revision*`.`*compare_mode*`.`*extension*
```
* *test-name* cannot contain dots. This is so that the general form of test
- *test-name* cannot contain dots. This is so that the general form of test
output filenames have a predictable form we can pattern match on in order to
track stray test output files.
* *revision* is the [revision](#cfg-revisions) name.
This is not included when not using revisions.
* *compare_mode* is the [compare mode](#compare-modes).
This will only be checked when the given compare mode is active.
If the file does not exist, then compiletest will check for a file without
the compare mode.
* *extension* is the kind of output being checked:
* `stderr` — compiler stderr
* `stdout` — compiler stdout
* `run.stderr` — stderr when running the test
* `run.stdout` — stdout when running the test
* `64bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 64-bit target
* `32bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 32-bit target
- *revision* is the [revision](#cfg-revisions) name. This is not included when
not using revisions.
- *compare_mode* is the [compare mode](#compare-modes). This will only be
checked when the given compare mode is active. If the file does not exist,
then compiletest will check for a file without the compare mode.
- *extension* is the kind of output being checked:
- `stderr` — compiler stderr
- `stdout` — compiler stdout
- `run.stderr` — stderr when running the test
- `run.stdout` — stdout when running the test
- `64bit.stderr` — compiler stderr with `stderr-per-bitwidth` directive on a
64-bit target
- `32bit.stderr` — compiler stderr with `stderr-per-bitwidth` directive on a
32-bit target
A simple example would be `foo.stderr` next to a `foo.rs` test.
A more complex example would be `foo.my-revision.polonius.stderr`.
There are several [headers](headers.md) which will change how compiletest will
check for output files:
There are several [directives](directives.md) which will change how compiletest
will check for output files:
* `stderr-per-bitwidth` — checks separate output files based on the target
pointer width. Consider using the `normalize-stderr` header instead (see
- `stderr-per-bitwidth` — checks separate output files based on the target
pointer width. Consider using the `normalize-stderr` directive instead (see
[Normalization](#normalization)).
* `dont-check-compiler-stderr` — Ignores stderr from the compiler.
* `dont-check-compiler-stdout` — Ignores stdout from the compiler.
* `compare-output-lines-by-subset` — Checks that the output contains the
- `dont-check-compiler-stderr` — Ignores stderr from the compiler.
- `dont-check-compiler-stdout` — Ignores stdout from the compiler.
- `compare-output-lines-by-subset` — Checks that the output contains the
contents of the stored output files by lines opposed to checking for strict
equality.
UI tests run with `-Zdeduplicate-diagnostics=no` flag which disables
rustc's built-in diagnostic deduplication mechanism.
This means you may see some duplicate messages in the output.
This helps illuminate situations where duplicate diagnostics are being
generated.
UI tests run with `-Zdeduplicate-diagnostics=no` flag which disables rustc's
built-in diagnostic deduplication mechanism. This means you may see some
duplicate messages in the output. This helps illuminate situations where
duplicate diagnostics are being generated.
### Normalization
@ -109,20 +102,20 @@ platforms, mainly about filenames.
Compiletest makes the following replacements on the compiler output:
- The directory where the test is defined is replaced with `$DIR`.
Example: `/path/to/rust/tests/ui/error-codes`
- The directory where the test is defined is replaced with `$DIR`. Example:
`/path/to/rust/tests/ui/error-codes`
- The directory to the standard library source is replaced with `$SRC_DIR`.
Example: `/path/to/rust/library`
- Line and column numbers for paths in `$SRC_DIR` are replaced with `LL:COL`.
This helps ensure that changes to the layout of the standard library do not
cause widespread changes to the `.stderr` files.
Example: `$SRC_DIR/alloc/src/sync.rs:53:46`
- The base directory where the test's output goes is replaced with `$TEST_BUILD_DIR`.
This only comes up in a few rare circumstances.
Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui`
cause widespread changes to the `.stderr` files. Example:
`$SRC_DIR/alloc/src/sync.rs:53:46`
- The base directory where the test's output goes is replaced with
`$TEST_BUILD_DIR`. This only comes up in a few rare circumstances. Example:
`/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui`
- Tabs are replaced with `\t`.
- Backslashes (`\`) are converted to forward slashes (`/`) within paths (using
a heuristic). This helps normalize differences with Windows-style paths.
- Backslashes (`\`) are converted to forward slashes (`/`) within paths (using a
heuristic). This helps normalize differences with Windows-style paths.
- CRLF newlines are converted to LF.
- Error line annotations like `//~ ERROR some message` are removed.
- Various v0 and legacy symbol hashes are replaced with placeholders like
@ -131,20 +124,20 @@ Compiletest makes the following replacements on the compiler output:
Additionally, the compiler is run with the `-Z ui-testing` flag which causes
the compiler itself to apply some changes to the diagnostic output to make it
more suitable for UI testing.
For example, it will anonymize line numbers in the output (line numbers
prefixing each source line are replaced with `LL`).
In extremely rare situations, this mode can be disabled with the header
command `//@ compile-flags: -Z ui-testing=no`.
prefixing each source line are replaced with `LL`). In extremely rare
situations, this mode can be disabled with the directive `//@
compile-flags: -Z ui-testing=no`.
Note: The line and column numbers for `-->` lines pointing to the test are
*not* normalized, and left as-is. This ensures that the compiler continues
to point to the correct location, and keeps the stderr files readable.
Ideally all line/column information would be retained, but small changes to
the source causes large diffs, and more frequent merge conflicts and test
errors.
Note: The line and column numbers for `-->` lines pointing to the test are *not*
normalized, and left as-is. This ensures that the compiler continues to point to
the correct location, and keeps the stderr files readable. Ideally all
line/column information would be retained, but small changes to the source
causes large diffs, and more frequent merge conflicts and test errors.
Sometimes these built-in normalizations are not enough. In such cases, you
may provide custom normalization rules using the header commands, e.g.
Sometimes these built-in normalizations are not enough. In such cases, you may
provide custom normalization rules using `normalize-*` directives, e.g.
```rust,ignore
//@ normalize-stdout-test: "foo" -> "bar"
@ -152,10 +145,10 @@ may provide custom normalization rules using the header commands, e.g.
//@ normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)"
```
This tells the test, on 32-bit platforms, whenever the compiler writes
`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)`
instead. Similar for 64-bit. The replacement is performed by regexes using
default regex flavor provided by `regex` crate.
This tells the test, on 32-bit platforms, whenever the compiler writes `fn() (32
bits)` to stderr, it should be normalized to read `fn() ($PTR bits)` instead.
Similar for 64-bit. The replacement is performed by regexes using default regex
flavor provided by `regex` crate.
The corresponding reference file will use the normalized output to test both
32-bit and 64-bit platforms:
@ -168,22 +161,21 @@ The corresponding reference file will use the normalized output to test both
...
```
Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`] for a
concrete usage example.
Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`] for a concrete
usage example.
[mrs]: https://github.com/rust-lang/rust/blob/master/tests/ui/transmute/main.rs
[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/tests/ui/transmute/main.stderr
Besides `normalize-stderr-32bit` and `-64bit`, one may use any target
information or stage supported by [`ignore-X`](headers.md#ignoring-tests)
here as well (e.g. `normalize-stderr-windows` or simply
`normalize-stderr-test` for unconditional replacement).
information or stage supported by [`ignore-X`](directives.md#ignoring-tests) here
as well (e.g. `normalize-stderr-windows` or simply `normalize-stderr-test` for
unconditional replacement).
## Error annotations
Error annotations specify the errors that the compiler is expected to emit.
They are "attached" to the line in source where the error is located.
Error annotations specify the errors that the compiler is expected to emit. They
are "attached" to the line in source where the error is located.
```rust,ignore
fn main() {
@ -191,41 +183,46 @@ fn main() {
}
```
Although UI tests have a `.stderr` file which contains the entire compiler output,
UI tests require that errors are also annotated within the source.
This redundancy helps avoid mistakes since the `.stderr` files are usually
auto-generated.
It also helps to directly see where the error spans are expected to point to
by looking at one file instead of having to compare the `.stderr` file with
the source.
Finally, they ensure that no additional unexpected errors are generated.
Although UI tests have a `.stderr` file which contains the entire compiler
output, UI tests require that errors are also annotated within the source. This
redundancy helps avoid mistakes since the `.stderr` files are usually
auto-generated. It also helps to directly see where the error spans are expected
to point to by looking at one file instead of having to compare the `.stderr`
file with the source. Finally, they ensure that no additional unexpected errors
are generated.
They have several forms, but generally are a comment with the diagnostic
level (such as `ERROR`) and a substring of the expected error output.
You don't have to write out the entire message, just make sure to include the
important part of the message to make it self-documenting.
They have several forms, but generally are a comment with the diagnostic level
(such as `ERROR`) and a substring of the expected error output. You don't have
to write out the entire message, just make sure to include the important part of
the message to make it self-documenting.
The error annotation needs to match with the line of the diagnostic.
There are several ways to match the message with the line (see the examples below):
The error annotation needs to match with the line of the diagnostic. There are
several ways to match the message with the line (see the examples below):
* `~`: Associates the error level and message with the current line
* `~^`: Associates the error level and message with the previous error
annotation line.
Each caret (`^`) that you add adds a line to this, so `~^^^` is three lines
above the error annotation line.
* `~|`: Associates the error level and message with the same line as the
previous comment.
This is more convenient than using multiple carets when there are multiple
messages associated with the same line.
* `~`: Associates the error level and message with the *current* line
* `~^`: Associates the error level and message with the *previous* error
annotation line. Each caret (`^`) that you add adds a line to this, so `~^^^`
is three lines above the error annotation line.
* `~|`: Associates the error level and message with the *same* line as the
*previous comment*. This is more convenient than using multiple carets when
there are multiple messages associated with the same line.
The space character between `//~` (or other variants) and the subsequent text
is negligible (i.e. there is no semantic difference between `//~ ERROR` and
Example:
```rust,ignore
let _ = same_line; //~ ERROR undeclared variable
fn meow(_: [u8]) {}
//~^ ERROR unsized
//~| ERROR anonymous parameters
```
The space character between `//~` (or other variants) and the subsequent text is
negligible (i.e. there is no semantic difference between `//~ ERROR` and
`//~ERROR` although the former is more common in the codebase).
### Error annotation examples
Here are examples of error annotations on different lines of UI test
source.
Here are examples of error annotations on different lines of UI test source.
#### Positioned on error line
@ -243,10 +240,9 @@ fn main() {
#### Positioned below error line
Use the `//~^` idiom with number of carets in the string to indicate the
number of lines above.
In the example below, the error line is four lines above the error annotation
line so four carets are included in the annotation.
Use the `//~^` idiom with number of carets in the string to indicate the number
of lines above. In the example below, the error line is four lines above the
error annotation line so four carets are included in the annotation.
```rust,ignore
fn main() {
@ -280,8 +276,8 @@ fn main() {
### `error-pattern`
The `error-pattern` [header](headers.md) can be used for
messages that don't have a specific span.
The `error-pattern` [directive](directives.md) can be used for messages that don't
have a specific span.
Let's think about this test:
@ -294,9 +290,9 @@ fn main() {
}
```
We want to ensure this shows "index out of bounds" but we cannot use the
`ERROR` annotation since the error doesn't have any span.
Then it's time to use the `error-pattern` header:
We want to ensure this shows "index out of bounds" but we cannot use the `ERROR`
annotation since the error doesn't have any span. Then it's time to use the
`error-pattern` directive:
```rust,ignore
//@ error-pattern: index out of bounds
@ -314,33 +310,32 @@ But for strict testing, try to use the `ERROR` annotation as much as possible.
The error levels that you can have are:
1. `ERROR`
2. `WARN` or `WARNING`
3. `NOTE`
4. `HELP` and `SUGGESTION`
- `ERROR`
- `WARN` or `WARNING`
- `NOTE`
- `HELP` and `SUGGESTION`
You are allowed to not include a level, but you should include it at least for
the primary message.
The `SUGGESTION` level is used for specifying what the expected replacement
text should be for a diagnostic suggestion.
The `SUGGESTION` level is used for specifying what the expected replacement text
should be for a diagnostic suggestion.
UI tests use the `-A unused` flag by default to ignore all unused warnings, as
unused warnings are usually not the focus of a test.
However, simple code samples often have unused warnings.
If the test is specifically testing an unused warning, just add the
appropriate `#![warn(unused)]` attribute as needed.
unused warnings are usually not the focus of a test. However, simple code
samples often have unused warnings. If the test is specifically testing an
unused warning, just add the appropriate `#![warn(unused)]` attribute as needed.
### cfg revisions
### `cfg` revisions
When using [revisions](compiletest.md#revisions), different messages can be
conditionally checked based on the current revision.
This is done by placing the revision cfg name in brackets like this:
conditionally checked based on the current revision. This is done by placing the
revision cfg name in brackets like this:
```rust,ignore
//@ edition:2018
//@ revisions: mir thir
//@ [thir]compile-flags: -Z thir-unsafeck
//@[thir] compile-flags: -Z thir-unsafeck
async unsafe fn f() {}
@ -356,10 +351,10 @@ fn main() {
In this example, the second error message is only emitted in the `mir` revision.
The `thir` revision only emits the first error.
If the cfg causes the compiler to emit different output, then a test can have
multiple `.stderr` files for the different outputs.
In the example above, there would be a `.mir.stderr` and `.thir.stderr` file
with the different outputs of the different revisions.
If the `cfg` causes the compiler to emit different output, then a test can have
multiple `.stderr` files for the different outputs. In the example above, there
would be a `.mir.stderr` and `.thir.stderr` file with the different outputs of
the different revisions.
> Note: cfg revisions also work inside the source code with `#[cfg]` attributes.
>
@ -368,113 +363,108 @@ with the different outputs of the different revisions.
## Controlling pass/fail expectations
By default, a UI test is expected to **generate a compile error** because most
of the tests are checking for invalid input and error diagnostics.
However, you can also make UI tests where compilation is expected to succeed,
and you can even run the resulting program.
Just add one of the following [header commands](headers.md):
of the tests are checking for invalid input and error diagnostics. However, you
can also make UI tests where compilation is expected to succeed, and you can
even run the resulting program. Just add one of the following
[directives](directives.md):
* Pass headers:
* `//@ check-pass` — compilation should succeed but skip codegen
- Pass directives:
- `//@ check-pass` — compilation should succeed but skip codegen
(which is expensive and isn't supposed to fail in most cases).
* `//@ build-pass` — compilation and linking should succeed but do
- `//@ build-pass` — compilation and linking should succeed but do
not run the resulting binary.
* `//@ run-pass` — compilation should succeed and running the resulting
- `//@ run-pass` — compilation should succeed and running the resulting
binary should also succeed.
* Fail headers:
* `//@ check-fail` — compilation should fail (the codegen phase is skipped).
- Fail directives:
- `//@ check-fail` — compilation should fail (the codegen phase is skipped).
This is the default for UI tests.
* `//@ build-fail` — compilation should fail during the codegen phase.
- `//@ build-fail` — compilation should fail during the codegen phase.
This will run `rustc` twice, once to verify that it compiles successfully
without the codegen phase, then a second time the full compile should
fail.
* `//@ run-fail` — compilation should succeed, but running the resulting
- `//@ run-fail` — compilation should succeed, but running the resulting
binary should fail.
For `run-pass` and `run-fail` tests, by default the output of the program
itself is not checked.
If you want to check the output of running the program, include the
`check-run-results` header.
This will check for a `.run.stderr` and `.run.stdout` files to compare
against the actual output of the program.
For `run-pass` and `run-fail` tests, by default the output of the program itself
is not checked.
Tests with the `*-pass` headers can be overridden with the `--pass`
If you want to check the output of running the program, include the
`check-run-results` directive. This will check for a `.run.stderr` and
`.run.stdout` files to compare against the actual output of the program.
Tests with the `*-pass` directives can be overridden with the `--pass`
command-line option:
```sh
./x test tests/ui --pass check
```
The `--pass` option only affects UI tests.
Using `--pass check` can run the UI test suite much faster (roughly twice as
fast on my system), though obviously not exercising as much.
The `--pass` option only affects UI tests. Using `--pass check` can run the UI
test suite much faster (roughly twice as fast on my system), though obviously
not exercising as much.
The `ignore-pass` header can be used to ignore the `--pass` CLI flag if the
The `ignore-pass` directive can be used to ignore the `--pass` CLI flag if the
test won't work properly with that override.
## Known bugs
The `known-bug` header may be used for tests that demonstrate a known bug that
has not yet been fixed.
Adding tests for known bugs is helpful for several reasons, including:
The `known-bug` directive may be used for tests that demonstrate a known bug
that has not yet been fixed. Adding tests for known bugs is helpful for several
reasons, including:
1. Maintaining a functional test that can be conveniently reused when the bug is fixed.
2. Providing a sentinel that will fail if the bug is incidentally fixed.
This can alert the developer so they know that the associated issue has
been fixed and can possibly be closed.
1. Maintaining a functional test that can be conveniently reused when the bug is
fixed.
2. Providing a sentinel that will fail if the bug is incidentally fixed. This
can alert the developer so they know that the associated issue has been fixed
and can possibly be closed.
Do not include [error annotations](#error-annotations) in a test with `known-bug`.
The test should still include other normal headers and stdout/stderr files.
Do not include [error annotations](#error-annotations) in a test with
`known-bug`. The test should still include other normal directives and
stdout/stderr files.
## Test organization
When deciding where to place a test file, please try to find a subdirectory
that best matches what you are trying to exercise.
Do your best to keep things organized.
Admittedly it can be difficult as some tests can overlap different categories,
and the existing layout may not fit well.
When deciding where to place a test file, please try to find a subdirectory that
best matches what you are trying to exercise. Do your best to keep things
organized. Admittedly it can be difficult as some tests can overlap different
categories, and the existing layout may not fit well.
For regression tests – basically, some random snippet of code that came in
from the internet – we often name the test after the issue plus a short
description.
Ideally, the test should be added to a directory that helps identify what
piece of code is being tested here (e.g.,
`tests/ui/borrowck/issue-54597-reject-move-out-of-borrow-via-pat.rs`)
Name the test by a concise description of what the test is checking. Avoid
including the issue number in the test name. See [best
practices](best-practices.md) for a more in-depth discussion of this.
When writing a new feature, **create a subdirectory to store your tests**.
For example, if you are implementing RFC 1234 ("Widgets"), then it might make
sense to put the tests in a directory like `tests/ui/rfc1234-widgets/`.
Ideally, the test should be added to a directory that helps identify what piece
of code is being tested here (e.g.,
`tests/ui/borrowck/reject-move-out-of-borrow-via-pat.rs`)
In other cases, there may already be a suitable directory. (The proper
directory structure to use is actually an area of active debate.)
When writing a new feature, you may want to **create a subdirectory to store
your tests**. For example, if you are implementing RFC 1234 ("Widgets"), then it
might make sense to put the tests in a directory like
`tests/ui/rfc1234-widgets/`.
Over time, the [`tests/ui`] directory has grown very fast.
There is a check in [tidy](intro.md#tidy) that will ensure none of the
subdirectories has more than 1000 entries.
Having too many files causes problems because it isn't editor/IDE friendly and
the GitHub UI won't show more than 1000 entries.
However, since `tests/ui` (UI test root directory) and `tests/ui/issues`
directories have more than 1000 entries, we set a different limit for those
directories.
So, please avoid putting a new test there and try to find a more relevant
place.
In other cases, there may already be a suitable directory.
Over time, the [`tests/ui`] directory has grown very fast. There is a check in
[tidy](intro.md#tidy) that will ensure none of the subdirectories has more than
1000 entries. Having too many files causes problems because it isn't editor/IDE
friendly and the GitHub UI won't show more than 1000 entries. However, since
`tests/ui` (UI test root directory) and `tests/ui/issues` directories have more
than 1000 entries, we set a different limit for those directories. So, please
avoid putting a new test there and try to find a more relevant place.
For example, if your test is related to closures, you should put it in
`tests/ui/closures`.
If you're not sure where is the best place, it's still okay to add to
`tests/ui/issues/`.
When you reach the limit, you could increase it by tweaking [here][ui test
tidy].
`tests/ui/closures`. When you reach the limit, you could increase it by tweaking
[here][ui test tidy].
[ui test tidy]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/ui_tests.rs
## Rustfix tests
UI tests can validate that diagnostic suggestions apply correctly
and that the resulting changes compile correctly.
This can be done with the `run-rustfix` header:
UI tests can validate that diagnostic suggestions apply correctly and that the
resulting changes compile correctly. This can be done with the `run-rustfix`
directive:
```rust,ignore
//@ run-rustfix
@ -487,40 +477,40 @@ pub struct not_camel_case {}
//~| SUGGESTION NotCamelCase
```
Rustfix tests should have a file with the `.fixed` extension which contains
the source file after the suggestion has been applied.
Rustfix tests should have a file with the `.fixed` extension which contains the
source file after the suggestion has been applied.
When the test is run, compiletest first checks that the correct
lint/warning is generated.
Then, it applies the suggestion and compares against `.fixed` (they must match).
Finally, the fixed source is compiled, and this compilation is required to succeed.
- When the test is run, compiletest first checks that the correct lint/warning
is generated.
- Then, it applies the suggestion and compares against `.fixed` (they must
match).
- Finally, the fixed source is compiled, and this compilation is required to
succeed.
Usually when creating a rustfix test you will generate the `.fixed` file
automatically with the `x test --bless` option.
The `run-rustfix` header will cause *all* suggestions to be applied, even
if they are not [`MachineApplicable`](../diagnostics.md#suggestions).
If this is a problem, then you can add the `rustfix-only-machine-applicable`
header in addition to `run-rustfix`.
This should be used if there is a mixture of different suggestion levels, and
some of the non-machine-applicable ones do not apply cleanly.
The `run-rustfix` directive will cause *all* suggestions to be applied, even if
they are not [`MachineApplicable`](../diagnostics.md#suggestions). If this is a
problem, then you can add the `rustfix-only-machine-applicable` directive in
addition to `run-rustfix`. This should be used if there is a mixture of
different suggestion levels, and some of the non-machine-applicable ones do not
apply cleanly.
## Compare modes
[Compare modes](compiletest.md#compare-modes) can be used to run all tests
with different flags from what they are normally compiled with.
In some cases, this might result in different output from the compiler.
To support this, different output files can be saved which contain the
output based on the compare mode.
[Compare modes](compiletest.md#compare-modes) can be used to run all tests with
different flags from what they are normally compiled with. In some cases, this
might result in different output from the compiler. To support this, different
output files can be saved which contain the output based on the compare mode.
For example, when using the Polonius mode, a test `foo.rs` will
first look for expected output in `foo.polonius.stderr`, falling back to the usual
`foo.stderr` if not found.
This is useful as different modes can sometimes result in different
diagnostics and behavior.
This can help track which tests have differences between the modes, and to
visually inspect those diagnostic differences.
For example, when using the Polonius mode, a test `foo.rs` will first look for
expected output in `foo.polonius.stderr`, falling back to the usual `foo.stderr`
if not found. This is useful as different modes can sometimes result in
different diagnostics and behavior. This can help track which tests have
differences between the modes, and to visually inspect those diagnostic
differences.
If in the rare case you encounter a test that has different behavior, you can
run something like the following to generate the alternate stderr file:
@ -533,13 +523,15 @@ Currently none of the compare modes are checked in CI for UI tests.
## `rustc_*` TEST attributes
The compiler defines several perma-unstable `#[rustc_*]` attributes gated behind the internal feature
`rustc_attrs` that dump extra compiler-internal information. See the corresponding subsection in
[compiler debugging] for more details.
The compiler defines several perma-unstable `#[rustc_*]` attributes gated behind
the internal feature `rustc_attrs` that dump extra compiler-internal
information. See the corresponding subsection in [compiler debugging] for more
details.
They can be used in tests to more precisely, legibly and easily test internal compiler state in cases
where it would otherwise be very hard to do the same with "user-facing" Rust alone. Indeed, one could
say that this slightly abuses the term "UI" (*user* interface) and turns such UI tests from black-box
tests into white-box ones. Use them carefully and sparingly.
They can be used in tests to more precisely, legibly and easily test internal
compiler state in cases where it would otherwise be very hard to do the same
with "user-facing" Rust alone. Indeed, one could say that this slightly abuses
the term "UI" (*user* interface) and turns such UI tests from black-box tests
into white-box ones. Use them carefully and sparingly.
[compiler debugging]: ../compiler-debugging.md#rustc_test-attributes