[Testing 1/2] Revise testing chapters excluding the directives chapter (#2088)
This commit is contained in:
parent
0a9b996970
commit
2b9274252f
|
|
@ -21,10 +21,11 @@
|
||||||
- [Testing with Docker](./tests/docker.md)
|
- [Testing with Docker](./tests/docker.md)
|
||||||
- [Testing with CI](./tests/ci.md)
|
- [Testing with CI](./tests/ci.md)
|
||||||
- [Adding new tests](./tests/adding.md)
|
- [Adding new tests](./tests/adding.md)
|
||||||
|
- [Best practices](./tests/best-practices.md)
|
||||||
- [Compiletest](./tests/compiletest.md)
|
- [Compiletest](./tests/compiletest.md)
|
||||||
- [UI tests](./tests/ui.md)
|
- [UI tests](./tests/ui.md)
|
||||||
- [Test headers](./tests/headers.md)
|
- [Test headers](./tests/directives.md)
|
||||||
- [Integration testing](./tests/integration.md)
|
- [Ecosystem testing](./tests/ecosystem.md)
|
||||||
- [Crater](./tests/crater.md)
|
- [Crater](./tests/crater.md)
|
||||||
- [Fuchsia](./tests/fuchsia.md)
|
- [Fuchsia](./tests/fuchsia.md)
|
||||||
- [Rust for Linux](./tests/rust-for-linux.md)
|
- [Rust for Linux](./tests/rust-for-linux.md)
|
||||||
|
|
|
||||||
|
|
@ -2,55 +2,62 @@
|
||||||
|
|
||||||
<!-- toc -->
|
<!-- toc -->
|
||||||
|
|
||||||
**In general, we expect every PR that fixes a bug in rustc to come
|
**In general, we expect every PR that fixes a bug in rustc to come accompanied
|
||||||
accompanied by a regression test of some kind.** This test should fail
|
by a regression test of some kind.** This test should fail in master but pass
|
||||||
in master but pass after the PR. These tests are really useful for
|
after the PR. These tests are really useful for preventing us from repeating the
|
||||||
preventing us from repeating the mistakes of the past.
|
mistakes of the past.
|
||||||
|
|
||||||
The first thing to decide is which kind of test to add.
|
The first thing to decide is which kind of test to add. This will depend on the
|
||||||
This will depend on the nature of the change and what you want to exercise.
|
nature of the change and what you want to exercise. Here are some rough
|
||||||
Here are some rough guidelines:
|
guidelines:
|
||||||
|
|
||||||
- The majority of compiler tests are done with [compiletest].
|
- The majority of compiler tests are done with [compiletest].
|
||||||
- The majority of compiletest tests are [UI](ui.md) tests in the [`tests/ui`] directory.
|
- The majority of compiletest tests are [UI](ui.md) tests in the [`tests/ui`]
|
||||||
- Changes to the standard library are usually tested within the standard library itself.
|
directory.
|
||||||
- The majority of standard library tests are written as doctests,
|
- Changes to the standard library are usually tested within the standard library
|
||||||
which illustrate and exercise typical API behavior.
|
itself.
|
||||||
|
- The majority of standard library tests are written as doctests, which
|
||||||
|
illustrate and exercise typical API behavior.
|
||||||
- Additional [unit tests](intro.md#package-tests) should go in
|
- Additional [unit tests](intro.md#package-tests) should go in
|
||||||
`library/${crate}/tests` (where `${crate}` is usually `core`, `alloc`, or `std`).
|
`library/${crate}/tests` (where `${crate}` is usually `core`, `alloc`, or
|
||||||
- If the code is part of an isolated system, and you are not testing compiler output,
|
`std`).
|
||||||
consider using a [unit or integration test](intro.md#package-tests).
|
- If the code is part of an isolated system, and you are not testing compiler
|
||||||
- Need to run rustdoc? Prefer a `rustdoc` or `rustdoc-ui` test.
|
output, consider using a [unit or integration test](intro.md#package-tests).
|
||||||
Occasionally you'll need `rustdoc-js` as well.
|
- Need to run rustdoc? Prefer a `rustdoc` or `rustdoc-ui` test. Occasionally
|
||||||
|
you'll need `rustdoc-js` as well.
|
||||||
- Other compiletest test suites are generally used for special purposes:
|
- Other compiletest test suites are generally used for special purposes:
|
||||||
- Need to run gdb or lldb? Use the `debuginfo` test suite.
|
- Need to run gdb or lldb? Use the `debuginfo` test suite.
|
||||||
- Need to inspect LLVM IR or MIR IR? Use the `codegen` or `mir-opt` test suites.
|
- Need to inspect LLVM IR or MIR IR? Use the `codegen` or `mir-opt` test
|
||||||
- Need to inspect the resulting binary in some way?
|
suites.
|
||||||
Then use `run-make`.
|
- Need to inspect the resulting binary in some way? Or if all the other test
|
||||||
|
suites are too limited for your purposes? Then use `run-make`.
|
||||||
- Check out the [compiletest] chapter for more specialized test suites.
|
- Check out the [compiletest] chapter for more specialized test suites.
|
||||||
|
|
||||||
|
After deciding on which kind of test to add, see [best
|
||||||
|
practices](best-practices.md) for guidance on how to author tests that are easy
|
||||||
|
to work with that stand the test of time (i.e. if a test fails or need to be
|
||||||
|
modified several years later, how can we make it easier for them?).
|
||||||
|
|
||||||
[compiletest]: compiletest.md
|
[compiletest]: compiletest.md
|
||||||
[`tests/ui`]: https://github.com/rust-lang/rust/tree/master/tests/ui/
|
[`tests/ui`]: https://github.com/rust-lang/rust/tree/master/tests/ui/
|
||||||
|
|
||||||
## UI test walkthrough
|
## UI test walkthrough
|
||||||
|
|
||||||
The following is a basic guide for creating a [UI test](ui.md), which is one
|
The following is a basic guide for creating a [UI test](ui.md), which is one of
|
||||||
of the most common compiler tests.
|
the most common compiler tests. For this tutorial, we'll be adding a test for an
|
||||||
For this tutorial, we'll be adding a test for an async error message.
|
async error message.
|
||||||
|
|
||||||
### Step 1. Add a test file
|
### Step 1: Add a test file
|
||||||
|
|
||||||
The first step is to create a Rust source file somewhere in the
|
The first step is to create a Rust source file somewhere in the [`tests/ui`]
|
||||||
[`tests/ui`] tree.
|
tree. When creating a test, do your best to find a good location and name (see
|
||||||
When creating a test, do your best to find a good location and name (see [Test
|
[Test organization](ui.md#test-organization) for more). Since naming is the
|
||||||
organization](ui.md#test-organization) for more).
|
hardest part of development, everything should be downhill from here!
|
||||||
Since naming is the hardest part of development, everything should be downhill
|
|
||||||
from here!
|
|
||||||
|
|
||||||
Let's place our async test at `tests/ui/async-await/await-without-async.rs`:
|
Let's place our async test at `tests/ui/async-await/await-without-async.rs`:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
// Check what happens when using await in a non-async fn.
|
// Provide diagnostics when the user writes `await` in a non-`async` function.
|
||||||
//@ edition:2018
|
//@ edition:2018
|
||||||
|
|
||||||
async fn foo() {}
|
async fn foo() {}
|
||||||
|
|
@ -64,24 +71,22 @@ fn main() {}
|
||||||
|
|
||||||
A few things to notice about our test:
|
A few things to notice about our test:
|
||||||
|
|
||||||
* The top should start with a short comment that [explains what the test is
|
- The top should start with a short comment that [explains what the test is
|
||||||
for](#explanatory_comment).
|
for](#explanatory_comment).
|
||||||
* The `//@ edition:2018` comment is called a [header](headers.md) which provides
|
- The `//@ edition:2018` comment is called a [directive](directives.md) which
|
||||||
instructions to compiletest on how to build the test.
|
provides instructions to compiletest on how to build the test. Here we need to
|
||||||
Here we need to set the edition for `async` to work (the default is 2015).
|
set the edition for `async` to work (the default is edition 2015).
|
||||||
* Following that is the source of the test.
|
- Following that is the source of the test. Try to keep it succinct and to the
|
||||||
Try to keep it succinct and to the point.
|
point. This may require some effort if you are trying to minimize an example
|
||||||
This may require some effort if you are trying to minimize an example from a
|
from a bug report.
|
||||||
bug report.
|
- We end this test with an empty `fn main` function. This is because the default
|
||||||
* We end this test with an empty `fn main` function.
|
for UI tests is a `bin` crate-type, and we don't want the "main not found"
|
||||||
This is because the default for UI tests is a `bin` crate-type,
|
error in our test. Alternatively, you could add `#![crate_type="lib"]`.
|
||||||
and we don't want the "main not found" error in our test.
|
|
||||||
Alternatively, you could add `#![crate_type="lib"]`.
|
|
||||||
|
|
||||||
### Step 2. Generate the expected output
|
### Step 2: Generate the expected output
|
||||||
|
|
||||||
The next step is to create the expected output from the compiler.
|
The next step is to create the expected output snapshots from the compiler. This
|
||||||
This can be done with the `--bless` option:
|
can be done with the `--bless` option:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
./x test tests/ui/async-await/await-without-async.rs --bless
|
./x test tests/ui/async-await/await-without-async.rs --bless
|
||||||
|
|
@ -91,28 +96,30 @@ This will build the compiler (if it hasn't already been built), compile the
|
||||||
test, and place the output of the compiler in a file called
|
test, and place the output of the compiler in a file called
|
||||||
`tests/ui/async-await/await-without-async.stderr`.
|
`tests/ui/async-await/await-without-async.stderr`.
|
||||||
|
|
||||||
However, this step will fail!
|
However, this step will fail! You should see an error message, something like
|
||||||
You should see an error message, something like this:
|
this:
|
||||||
|
|
||||||
> error: /rust/tests/ui/async-await/await-without-async.rs:7: unexpected
|
> error: /rust/tests/ui/async-await/await-without-async.rs:7: unexpected
|
||||||
> error: '7:10: 7:16: `await` is only allowed inside `async` functions and
|
> error: '7:10: 7:16: `await` is only allowed inside `async` functions and
|
||||||
> blocks E0728'
|
> blocks E0728'
|
||||||
|
|
||||||
### Step 3. Add error annotations
|
This is because the stderr contains errors which were not matched by error
|
||||||
|
annotations in the source file.
|
||||||
|
|
||||||
Every error needs to be annotated with a comment in the source with the text
|
### Step 3: Add error annotations
|
||||||
of the error.
|
|
||||||
In this case, we can add the following comment to our test file:
|
Every error needs to be annotated with a comment in the source with the text of
|
||||||
|
the error. In this case, we can add the following comment to our test file:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
fn bar() {
|
fn bar() {
|
||||||
foo().await
|
foo().await
|
||||||
//~^ ERROR `await` is only allowed inside `async` functions and blocks
|
//~^ ERROR `await` is only allowed inside `async` functions and blocks
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The `//~^` squiggle caret comment tells compiletest that the error belongs to
|
The `//~^` squiggle caret comment tells compiletest that the error belongs to
|
||||||
the previous line (more on this in the [Error
|
the *previous* line (more on this in the [Error
|
||||||
annotations](ui.md#error-annotations) section).
|
annotations](ui.md#error-annotations) section).
|
||||||
|
|
||||||
Save that, and run the test again:
|
Save that, and run the test again:
|
||||||
|
|
@ -123,16 +130,15 @@ Save that, and run the test again:
|
||||||
|
|
||||||
It should now pass, yay!
|
It should now pass, yay!
|
||||||
|
|
||||||
### Step 4. Review the output
|
### Step 4: Review the output
|
||||||
|
|
||||||
Somewhat hand-in-hand with the previous step, you should inspect the `.stderr`
|
Somewhat hand-in-hand with the previous step, you should inspect the `.stderr`
|
||||||
file that was created to see if it looks like how you expect.
|
file that was created to see if it looks like how you expect. If you are adding
|
||||||
If you are adding a new diagnostic message, now would be a good time to
|
a new diagnostic message, now would be a good time to also consider how readable
|
||||||
also consider how readable the message looks overall, particularly for
|
the message looks overall, particularly for people new to Rust.
|
||||||
people new to Rust.
|
|
||||||
|
|
||||||
Our example `tests/ui/async-await/await-without-async.stderr` file should
|
Our example `tests/ui/async-await/await-without-async.stderr` file should look
|
||||||
look like this:
|
like this:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
error[E0728]: `await` is only allowed inside `async` functions and blocks
|
error[E0728]: `await` is only allowed inside `async` functions and blocks
|
||||||
|
|
@ -148,48 +154,47 @@ error: aborting due to previous error
|
||||||
For more information about this error, try `rustc --explain E0728`.
|
For more information about this error, try `rustc --explain E0728`.
|
||||||
```
|
```
|
||||||
|
|
||||||
You may notice some things look a little different than the regular
|
You may notice some things look a little different than the regular compiler
|
||||||
compiler output.
|
output.
|
||||||
The `$DIR` removes the path information which will differ between systems.
|
|
||||||
The `LL` values replace the line numbers.
|
- The `$DIR` removes the path information which will differ between systems.
|
||||||
That helps avoid small changes in the source from triggering large diffs.
|
- The `LL` values replace the line numbers. That helps avoid small changes in
|
||||||
See the [Normalization](ui.md#normalization) section for more.
|
the source from triggering large diffs. See the
|
||||||
|
[Normalization](ui.md#normalization) section for more.
|
||||||
|
|
||||||
Around this stage, you may need to iterate over the last few steps a few times
|
Around this stage, you may need to iterate over the last few steps a few times
|
||||||
to tweak your test, re-bless the test, and re-review the output.
|
to tweak your test, re-bless the test, and re-review the output.
|
||||||
|
|
||||||
### Step 5. Check other tests
|
### Step 5: Check other tests
|
||||||
|
|
||||||
Sometimes when adding or changing a diagnostic message, this will affect
|
Sometimes when adding or changing a diagnostic message, this will affect other
|
||||||
other tests in the test suite.
|
tests in the test suite. The final step before posting a PR is to check if you
|
||||||
The final step before posting a PR is to check if you have affected anything else.
|
have affected anything else. Running the UI suite is usually a good start:
|
||||||
Running the UI suite is usually a good start:
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
./x test tests/ui
|
./x test tests/ui
|
||||||
```
|
```
|
||||||
|
|
||||||
If other tests start failing, you may need to investigate what has changed
|
If other tests start failing, you may need to investigate what has changed and
|
||||||
and if the new output makes sense.
|
if the new output makes sense.
|
||||||
|
|
||||||
You may also need to re-bless the output with the `--bless` flag.
|
You may also need to re-bless the output with the `--bless` flag.
|
||||||
|
|
||||||
<a name="explanatory_comment"></a>
|
<a name="explanatory_comment"></a>
|
||||||
|
|
||||||
## Comment explaining what the test is about
|
## Comment explaining what the test is about
|
||||||
|
|
||||||
The first comment of a test file should **summarize the point
|
The first comment of a test file should **summarize the point of the test**, and
|
||||||
of the test**, and highlight what is important about it.
|
highlight what is important about it. If there is an issue number associated
|
||||||
If there is an issue number associated with the test, include
|
with the test, include the issue number.
|
||||||
the issue number.
|
|
||||||
|
|
||||||
This comment doesn't have to be super extensive. Just something like
|
This comment doesn't have to be super extensive. Just something like "Regression
|
||||||
"Regression test for #18060: match arms were matching in the wrong
|
test for #18060: match arms were matching in the wrong order." might already be
|
||||||
order." might already be enough.
|
enough.
|
||||||
|
|
||||||
These comments are very useful to others later on when your test
|
These comments are very useful to others later on when your test breaks, since
|
||||||
breaks, since they often can highlight what the problem is. They are
|
they often can highlight what the problem is. They are also useful if for some
|
||||||
also useful if for some reason the tests need to be refactored, since
|
reason the tests need to be refactored, since they let others know which parts
|
||||||
they let others know which parts of the test were important (often a
|
of the test were important. Often a test must be rewritten because it no longer
|
||||||
test must be rewritten because it no longer tests what is was meant to
|
tests what is was meant to test, and then it's useful to know what it *was*
|
||||||
test, and then it's useful to know what it *was* meant to test
|
meant to test exactly.
|
||||||
exactly).
|
|
||||||
|
|
|
||||||
|
|
@ -0,0 +1,188 @@
|
||||||
|
# Best practices for writing tests
|
||||||
|
|
||||||
|
This chapter describes best practices related to authoring and modifying tests.
|
||||||
|
We want to make sure the tests we author are easy to understand and modify, even
|
||||||
|
several years later, without needing to consult the original author and perform
|
||||||
|
a bunch of git archeology.
|
||||||
|
|
||||||
|
It's good practice to review the test that you authored by pretending that you
|
||||||
|
are a different contributor who is looking at the test that failed several years
|
||||||
|
later without much context (this also helps yourself even a few days or months
|
||||||
|
later!). Then ask yourself: how can I make my life and their lives easier?
|
||||||
|
|
||||||
|
To help put this into perspective, let's start with an aside on how to write a
|
||||||
|
test that makes the life of another contributor as hard as possible.
|
||||||
|
|
||||||
|
> **Aside: Simple Test Sabotage Field Manual**
|
||||||
|
>
|
||||||
|
> To make the life of another contributor as hard as possible, one might:
|
||||||
|
>
|
||||||
|
> - Only name the test after an issue, e.g. `issue-123456.rs`.
|
||||||
|
> - Have no comments at all on what the test is trying to exercise, no links to
|
||||||
|
> relevant context.
|
||||||
|
> - Include a test that is massive (that can otherwise be minimized) and
|
||||||
|
> contains non-essential pieces which distracts from the core thing the test
|
||||||
|
> is actually trying to test.
|
||||||
|
> - Include a bunch of unrelated syntax errors and other errors which are not
|
||||||
|
> critical to what the test is trying to check.
|
||||||
|
> - Weirdly format the snippets.
|
||||||
|
> - Include a bunch of unused and unrelated features.
|
||||||
|
> - Have e.g. `ignore-windows` [compiletest directives] but don't offer any
|
||||||
|
> explanation as to *why* they are needed.
|
||||||
|
|
||||||
|
## Test naming
|
||||||
|
|
||||||
|
Make it easy for the reader to immediately understand what the test is
|
||||||
|
exercising, instead of having to type in the issue number and dig through github
|
||||||
|
search for what the test is trying to exercise. This has an additional benefit
|
||||||
|
of making the test possible to be filtered via `--test-args` as a collection of
|
||||||
|
related tests.
|
||||||
|
|
||||||
|
- Name the test after what it's trying to exercise or prevent regressions of.
|
||||||
|
- Keep it concise.
|
||||||
|
- Avoid including issue numbers in test names.
|
||||||
|
|
||||||
|
> **Avoid issue numbers in test names**
|
||||||
|
>
|
||||||
|
> Prefer including them as links or `#123456` in test comments instead.
|
||||||
|
>
|
||||||
|
> ```text
|
||||||
|
> tests/ui/typeck/issue-123456.rs // bad
|
||||||
|
> tests/ui/typeck/issue-123456-asm-macro-external-span-ice.rs // bad
|
||||||
|
> tests/ui/typeck/asm-macro-external-span-ice.rs // good
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> `issue-123456.rs` does not tell you immediately anything about what the test
|
||||||
|
> is actually exercising meaning you need to do additional searching. Including
|
||||||
|
> the issue number in the test name is really noisy for finding relevant tests
|
||||||
|
> by what they're exercising (if you `ls` a test directory and get a bunch of
|
||||||
|
> `issue-xxxxx` prefixes). We can link to the issue in a test comment.
|
||||||
|
>
|
||||||
|
> ```rs
|
||||||
|
> //! Check that `asm!` macro including nested macros that come from external
|
||||||
|
> //! crates do not lead to a codepoint boundary assertion ICE.
|
||||||
|
> //!
|
||||||
|
> //! Regression test for <https://github.com/rust-lang/rust/issues/123456>.
|
||||||
|
> ```
|
||||||
|
|
||||||
|
## Test organization
|
||||||
|
|
||||||
|
- For most test suites, try to find a semantically meaningful subdirectory to
|
||||||
|
home the test.
|
||||||
|
- E.g. for an implementation of RFC 2093 specifically, we can group a
|
||||||
|
collection of tests under `tests/ui/rfc-2093-infer-outlives/`. For the
|
||||||
|
directory name, include what the RFC is about.
|
||||||
|
- For the [`run-make`] test suite, each `rmake.rs` must be contained within an
|
||||||
|
immediate subdirectory under `tests/run-make/`. Further nesting is not
|
||||||
|
presently supported. Avoid including issue number in the directory name too,
|
||||||
|
include that info in a comment inside `rmake.rs`.
|
||||||
|
|
||||||
|
## Test descriptions
|
||||||
|
|
||||||
|
To help other contributors understand what the test is about if their changes
|
||||||
|
lead to the test failing, we should make sure a test has sufficient docs about
|
||||||
|
its intent/purpose, links to relevant context (incl. issue numbers or other
|
||||||
|
discussions) and possibly relevant resources (e.g. can be helpful to link to
|
||||||
|
Win32 APIs for specific behavior).
|
||||||
|
|
||||||
|
**Synopsis of a test with good comments**
|
||||||
|
|
||||||
|
```rust,ignore
|
||||||
|
//! Brief summary of what the test is exercising.
|
||||||
|
//! Example: Regression test for #123456: make sure coverage attribute don't ICE
|
||||||
|
//! when applied to non-items.
|
||||||
|
//!
|
||||||
|
//! Optional: Remarks on related tests/issues, external APIs/tools, crash
|
||||||
|
//! mechanism, how it's fixed, FIXMEs, limitations, etc.
|
||||||
|
//! Example: This test is like `tests/attrs/linkage.rs`, but this test is
|
||||||
|
//! specifically checking `#[coverage]` which exercises a different code
|
||||||
|
//! path. The ICE was triggered during attribute validation when we tried
|
||||||
|
//! to construct a `def_path_str` but only emitted the diagnostic when the
|
||||||
|
//! platform is windows, causing an ICE on unix.
|
||||||
|
//!
|
||||||
|
//! Links to relevant issues and discussions. Examples below:
|
||||||
|
//! Regression test for <https://github.com/rust-lang/rust/issues/123456>.
|
||||||
|
//! See also <https://github.com/rust-lang/rust/issues/101345>.
|
||||||
|
//! See discussion at <https://rust-lang.zulipchat.com/#narrow/stream/131828-t-compiler/topic/123456-example-topic>.
|
||||||
|
//! See [`clone(2)`].
|
||||||
|
//!
|
||||||
|
//! [`clone(2)`]: https://man7.org/linux/man-pages/man2/clone.2.html
|
||||||
|
|
||||||
|
//@ ignore-windows
|
||||||
|
// Reason: (why is this test ignored for windows? why not specifically
|
||||||
|
// windows-gnu or windows-msvc?)
|
||||||
|
|
||||||
|
// Optional: Summary of test cases: What positive cases are checked?
|
||||||
|
// What negative cases are checked? Any specific quirks?
|
||||||
|
|
||||||
|
fn main() {
|
||||||
|
#[coverage]
|
||||||
|
//~^ ERROR coverage attribute can only be applied to function items.
|
||||||
|
let _ = {
|
||||||
|
// Comment highlighting something that deserves reader attention.
|
||||||
|
fn foo() {}
|
||||||
|
};
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
For how much context/explanation is needed, it is up to the author and
|
||||||
|
reviewer's discretion. A good rule of thumb is non-trivial things exercised in
|
||||||
|
the test deserves some explanation to help other contributors to understand.
|
||||||
|
This may include remarks on:
|
||||||
|
|
||||||
|
- How an ICE can get triggered if it's quite elaborate.
|
||||||
|
- Related issues and tests (e.g. this test is like another test but is kept
|
||||||
|
separate because...).
|
||||||
|
- Platform-specific behaviors.
|
||||||
|
- Behavior of external dependencies and APIs: syscalls, linkers, tools,
|
||||||
|
environments and the likes.
|
||||||
|
|
||||||
|
## Test content
|
||||||
|
|
||||||
|
- Try to make sure the test is as minimal as possible.
|
||||||
|
- Minimize non-critical code and especially minimize unnecessary syntax and type
|
||||||
|
errors which can clutter stderr snapshots.
|
||||||
|
- Where possible, use semantically meaningful names (e.g. `fn
|
||||||
|
bare_coverage_attributes() {}`).
|
||||||
|
|
||||||
|
## Flaky tests
|
||||||
|
|
||||||
|
All tests need to strive to be reproducible and reliable. Flaky tests are the
|
||||||
|
worst kind of tests, arguably even worse than not having the test in the first
|
||||||
|
place.
|
||||||
|
|
||||||
|
- Flaky tests can fail in completely unrelated PRs which can confuse other
|
||||||
|
contributors and waste their time trying to figure out if test failure is
|
||||||
|
related.
|
||||||
|
- Flaky tests provide no useful information from its test results other than
|
||||||
|
it's flaky and not reliable: if a test passed but it's flakey, did I just get
|
||||||
|
lucky? if a test is flakey but it failed, was it just spurious?
|
||||||
|
- Flaky tests degrade confidence in the whole test suite. If a test suite can
|
||||||
|
randomly spuriously fail due to flaky tests, did the whole test suite pass or
|
||||||
|
did I just get lucky/unlucky?
|
||||||
|
- Flaky tests can randomly fail in full CI, wasting previous full CI resources.
|
||||||
|
|
||||||
|
## Compiletest directives
|
||||||
|
|
||||||
|
See [compiletest directives] for a listing of directives.
|
||||||
|
|
||||||
|
- For `ignore-*`/`needs-*`/`only-*` directives, unless extremely obvious,
|
||||||
|
provide a brief remark on why the directive is needed. E.g. `"//@ ignore-wasi
|
||||||
|
(wasi codegens the main symbol differently)"`.
|
||||||
|
|
||||||
|
## FileCheck best practices
|
||||||
|
|
||||||
|
See [LLVM FileCheck guide][FileCheck] for details.
|
||||||
|
|
||||||
|
- Avoid matching on specific register numbers or basic block numbers unless
|
||||||
|
they're special or critical for the test. Consider using patterns to match
|
||||||
|
them where suitable.
|
||||||
|
|
||||||
|
> **TODO**
|
||||||
|
>
|
||||||
|
> Pending concrete advice.
|
||||||
|
|
||||||
|
[compiletest]: ./compiletest.md
|
||||||
|
[compiletest directives]: ./directives.md
|
||||||
|
[`run-make`]: ./compiletest.md#run-make-tests
|
||||||
|
[FileCheck]: https://llvm.org/docs/CommandGuide/FileCheck.html
|
||||||
347
src/tests/ci.md
347
src/tests/ci.md
|
|
@ -1,33 +1,53 @@
|
||||||
# Testing with CI
|
# Testing with CI
|
||||||
|
|
||||||
The primary goal of our CI system is to ensure that the `master` branch of `rust-lang/rust` is always in a valid state and passes our test suite.
|
The primary goal of our CI system is to ensure that the `master` branch of
|
||||||
|
`rust-lang/rust` is always in a valid state and passes our test suite.
|
||||||
|
|
||||||
From a high-level point of view, when you open a pull request at `rust-lang/rust`,
|
From a high-level point of view, when you open a pull request at
|
||||||
the following will happen:
|
`rust-lang/rust`, the following will happen:
|
||||||
|
|
||||||
- A small [subset](#pull-request-builds) of tests and checks are run after each push to the PR. This should help catching common errors.
|
- A small [subset](#pull-request-builds) of tests and checks are run after each
|
||||||
|
push to the PR. This should help catching common errors.
|
||||||
- When the PR is approved, the [bors] bot enqueues the PR into a [merge queue].
|
- When the PR is approved, the [bors] bot enqueues the PR into a [merge queue].
|
||||||
- Once the PR gets to the front of the queue, bors will create a merge commit
|
- Once the PR gets to the front of the queue, bors will create a merge commit
|
||||||
and run the [full test suite](#auto-builds) on it. The merge commit either contains only one specific PR or it can be a ["rollup"](#rollups) which combines multiple PRs together, to save CI costs.
|
and run the [full test suite](#auto-builds) on it. The merge commit either
|
||||||
- Once the whole test suite finishes, two things can happen. Either CI fails with an error that needs to be addressed by the developer, or CI succeeds and the merge commit is then pushed to the `master` branch.
|
contains only one specific PR or it can be a ["rollup"](#rollups) which
|
||||||
|
combines multiple PRs together, to save CI costs.
|
||||||
|
- Once the whole test suite finishes, two things can happen. Either CI fails
|
||||||
|
with an error that needs to be addressed by the developer, or CI succeeds and
|
||||||
|
the merge commit is then pushed to the `master` branch.
|
||||||
|
|
||||||
If you want to modify what gets executed on CI, see [Modifying CI jobs](#modifying-ci-jobs).
|
If you want to modify what gets executed on CI, see [Modifying CI
|
||||||
|
jobs](#modifying-ci-jobs).
|
||||||
|
|
||||||
## CI workflow
|
## CI workflow
|
||||||
|
|
||||||
<!-- date-check: may 2024 -->
|
<!-- date-check: Oct 2024 -->
|
||||||
Our CI is primarily executed on [GitHub Actions], with a single workflow defined in [`.github/workflows/ci.yml`], which contains a bunch of steps that are unified for all CI jobs that we execute. When a commit is pushed to a corresponding branch or a PR, the workflow executes the [`calculate-job-matrix.py`] script, which dynamically generates the specific CI jobs that should be executed. This script uses the [`jobs.yml`] file as an input, which contains a declarative configuration of all our CI jobs.
|
|
||||||
|
|
||||||
> Almost all build steps shell out to separate scripts. This keeps the CI fairly platform independent (i.e., we are not overly reliant on GitHub Actions). GitHub Actions is only relied on for bootstrapping the CI process and for orchestrating the scripts that drive the process.
|
Our CI is primarily executed on [GitHub Actions], with a single workflow defined
|
||||||
|
in [`.github/workflows/ci.yml`], which contains a bunch of steps that are
|
||||||
|
unified for all CI jobs that we execute. When a commit is pushed to a
|
||||||
|
corresponding branch or a PR, the workflow executes the
|
||||||
|
[`calculate-job-matrix.py`] script, which dynamically generates the specific CI
|
||||||
|
jobs that should be executed. This script uses the [`jobs.yml`] file as an
|
||||||
|
input, which contains a declarative configuration of all our CI jobs.
|
||||||
|
|
||||||
In essence, all CI jobs run `./x test`, `./x dist` or some other command with different configurations, across various operating systems, targets and platforms. There are two broad categories of jobs that are executed, `dist` and non-`dist` jobs.
|
> Almost all build steps shell out to separate scripts. This keeps the CI fairly
|
||||||
|
> platform independent (i.e., we are not overly reliant on GitHub Actions).
|
||||||
|
> GitHub Actions is only relied on for bootstrapping the CI process and for
|
||||||
|
> orchestrating the scripts that drive the process.
|
||||||
|
|
||||||
- Dist jobs build a full release of the compiler for a specific platform, including
|
In essence, all CI jobs run `./x test`, `./x dist` or some other command with
|
||||||
all the tools we ship through rustup; Those builds are then uploaded to the
|
different configurations, across various operating systems, targets and
|
||||||
`rust-lang-ci2` S3 bucket and are available to be locally installed with the
|
platforms. There are two broad categories of jobs that are executed, `dist` and
|
||||||
[rustup-toolchain-install-master] tool; The same builds are also used for
|
non-`dist` jobs.
|
||||||
actual releases: our release process basically consists of copying those
|
|
||||||
artifacts from `rust-lang-ci2` to the production endpoint and signing them.
|
- Dist jobs build a full release of the compiler for a specific platform,
|
||||||
|
including all the tools we ship through rustup; Those builds are then uploaded
|
||||||
|
to the `rust-lang-ci2` S3 bucket and are available to be locally installed
|
||||||
|
with the [rustup-toolchain-install-master] tool. The same builds are also used
|
||||||
|
for actual releases: our release process basically consists of copying those
|
||||||
|
artifacts from `rust-lang-ci2` to the production endpoint and signing them.
|
||||||
- Non-dist jobs run our full test suite on the platform, and the test suite of
|
- Non-dist jobs run our full test suite on the platform, and the test suite of
|
||||||
all the tools we ship through rustup; The amount of stuff we test depends on
|
all the tools we ship through rustup; The amount of stuff we test depends on
|
||||||
the platform (for example some tests are run only on Tier 1 platforms), and
|
the platform (for example some tests are run only on Tier 1 platforms), and
|
||||||
|
|
@ -37,63 +57,155 @@ artifacts from `rust-lang-ci2` to the production endpoint and signing them.
|
||||||
Based on an input event (usually a push to a branch), we execute one of three
|
Based on an input event (usually a push to a branch), we execute one of three
|
||||||
kinds of builds (sets of jobs).
|
kinds of builds (sets of jobs).
|
||||||
|
|
||||||
|
1. PR builds
|
||||||
|
2. Auto builds
|
||||||
|
3. Try builds
|
||||||
|
|
||||||
[rustup-toolchain-install-master]: https://github.com/kennytm/rustup-toolchain-install-master
|
[rustup-toolchain-install-master]: https://github.com/kennytm/rustup-toolchain-install-master
|
||||||
|
|
||||||
### Pull Request builds
|
### Pull Request builds
|
||||||
|
|
||||||
After each push to a pull request, a set of `pr` jobs are executed. Currently, these execute the
|
After each push to a pull request, a set of `pr` jobs are executed. Currently,
|
||||||
`x86_64-gnu-llvm-X`, `x86_64-gnu-tools`, `mingw-check` and `mingw-check-tidy` jobs, all running on Linux. These execute a relatively short (~30 minutes) and lightweight test suite that should catch common issues. More specifically, they run a set of lints, they try to perform a cross-compile check build to Windows mingw (without producing any artifacts) and they test the compiler using a *system* version of LLVM. Unfortunately, it would take too many resources to run the full test suite for each commit on every PR.
|
these execute the `x86_64-gnu-llvm-X`, `x86_64-gnu-tools`, `mingw-check` and
|
||||||
|
`mingw-check-tidy` jobs, all running on Linux. These execute a relatively short
|
||||||
|
(~30 minutes) and lightweight test suite that should catch common issues. More
|
||||||
|
specifically, they run a set of lints, they try to perform a cross-compile check
|
||||||
|
build to Windows mingw (without producing any artifacts) and they test the
|
||||||
|
compiler using a *system* version of LLVM. Unfortunately, it would take too many
|
||||||
|
resources to run the full test suite for each commit on every PR.
|
||||||
|
|
||||||
PR jobs are defined in the `pr` section of [`jobs.yml`]. They run under the `rust-lang/rust` repository, and their results can be observed directly on the PR, in the "CI checks" section at the bottom of the PR page.
|
> **Note on doc comments**
|
||||||
|
>
|
||||||
|
> Note that PR CI as of Oct 2024 <!-- datecheck --> by default does not try to
|
||||||
|
> run `./x doc xxx`. This means that if you have any broken intradoc links that
|
||||||
|
> would lead to `./x doc xxx` failing, it will happen very late into the full
|
||||||
|
> merge queue CI pipeline.
|
||||||
|
>
|
||||||
|
> Thus, it is a good idea to run `./x doc xxx` locally for any doc comment
|
||||||
|
> changes to help catch these early.
|
||||||
|
|
||||||
|
PR jobs are defined in the `pr` section of [`jobs.yml`]. They run under the
|
||||||
|
`rust-lang/rust` repository, and their results can be observed directly on the
|
||||||
|
PR, in the "CI checks" section at the bottom of the PR page.
|
||||||
|
|
||||||
### Auto builds
|
### Auto builds
|
||||||
|
|
||||||
Before a commit can be merged into the `master` branch, it needs to pass our complete test suite. We call this an `auto` build. This build runs tens of CI jobs that exercise various tests across operating systems and targets.
|
Before a commit can be merged into the `master` branch, it needs to pass our
|
||||||
The full test suite is quite slow; it can take two hours or more until all the `auto` CI jobs finish.
|
complete test suite. We call this an `auto` build. This build runs tens of CI
|
||||||
|
jobs that exercise various tests across operating systems and targets. The full
|
||||||
|
test suite is quite slow; it can take two hours or more until all the `auto` CI
|
||||||
|
jobs finish.
|
||||||
|
|
||||||
Most platforms only run the build steps, some run a restricted set of tests, only a subset run the full suite of tests (see Rust's [platform tiers]).
|
Most platforms only run the build steps, some run a restricted set of tests,
|
||||||
|
only a subset run the full suite of tests (see Rust's [platform tiers]).
|
||||||
|
|
||||||
Auto jobs are defined in the `auto` section of [`jobs.yml`]. They are executed on the `auto` branch under the `rust-lang-ci/rust` repository[^rust-lang-ci] and their results can be seen [here](https://github.com/rust-lang-ci/rust/actions), although usually you will be notified of the result by a comment made by bors on the corresponding PR.
|
Auto jobs are defined in the `auto` section of [`jobs.yml`]. They are executed
|
||||||
|
on the `auto` branch under the `rust-lang-ci/rust` repository[^rust-lang-ci] and
|
||||||
|
their results can be seen [here](https://github.com/rust-lang-ci/rust/actions),
|
||||||
|
although usually you will be notified of the result by a comment made by bors on
|
||||||
|
the corresponding PR.
|
||||||
|
|
||||||
At any given time, at most a single `auto` build is being executed. Find out more [here](#merging-prs-serially-with-bors).
|
At any given time, at most a single `auto` build is being executed. Find out
|
||||||
|
more [here](#merging-prs-serially-with-bors).
|
||||||
|
|
||||||
[platform tiers]: https://forge.rust-lang.org/release/platform-support.html#rust-platform-support
|
[platform tiers]: https://forge.rust-lang.org/release/platform-support.html#rust-platform-support
|
||||||
|
|
||||||
[^rust-lang-ci]: The `auto` and `try` jobs run under the `rust-lang-ci` fork for historical reasons. This may change in the future.
|
[^rust-lang-ci]: The `auto` and `try` jobs run under the `rust-lang-ci` fork for
|
||||||
|
historical reasons. This may change in the future.
|
||||||
|
|
||||||
### Try builds
|
### Try builds
|
||||||
|
|
||||||
Sometimes we want to run a subset of the test suite on CI for a given PR, or build a set of compiler artifacts from that PR, without attempting to merge it. We call this a "try build". A try build is started after a user with the proper permissions posts a PR comment with the `@bors try` command.
|
Sometimes we want to run a subset of the test suite on CI for a given PR, or
|
||||||
|
build a set of compiler artifacts from that PR, without attempting to merge it.
|
||||||
|
We call this a "try build". A try build is started after a user with the proper
|
||||||
|
permissions posts a PR comment with the `@bors try` command.
|
||||||
|
|
||||||
There are several use-cases for try builds:
|
There are several use-cases for try builds:
|
||||||
|
|
||||||
- Run a set of performance benchmarks using our [rustc-perf] benchmark suite. For this, a working compiler build is needed, which can be generated with a try build that runs the [dist-x86_64-linux] CI job, which builds an optimized version of the compiler on Linux (this job is currently executed by default when you start a try build). To create a try build and schedule it for a performance benchmark, you can use the `@bors try @rust-timer queue` command combination.
|
- Run a set of performance benchmarks using our [rustc-perf] benchmark suite.
|
||||||
- Check the impact of the PR across the Rust ecosystem, using a [crater] run. Again, a working compiler build is needed for this, which can be produced by the [dist-x86_64-linux] CI job.
|
For this, a working compiler build is needed, which can be generated with a
|
||||||
- Run a specific CI job (e.g. Windows tests) on a PR, to quickly test if it passes the test suite executed by that job. You can select which CI jobs will be executed in the try build by adding up to 10 lines containing `try-job: <name of job>` to the PR description. All such specified jobs will be executed in the try build once the `@bors try` command is used on the PR. If no try jobs are specified in this way, the jobs defined in the `try` section of [`jobs.yml`] will be executed by default.
|
try build that runs the [dist-x86_64-linux] CI job, which builds an optimized
|
||||||
|
version of the compiler on Linux (this job is currently executed by default
|
||||||
|
when you start a try build). To create a try build and schedule it for a
|
||||||
|
performance benchmark, you can use the `@bors try @rust-timer queue` command
|
||||||
|
combination.
|
||||||
|
- Check the impact of the PR across the Rust ecosystem, using a [crater] run.
|
||||||
|
Again, a working compiler build is needed for this, which can be produced by
|
||||||
|
the [dist-x86_64-linux] CI job.
|
||||||
|
- Run a specific CI job (e.g. Windows tests) on a PR, to quickly test if it
|
||||||
|
passes the test suite executed by that job. You can select which CI jobs will
|
||||||
|
be executed in the try build by adding up to 10 lines containing `try-job:
|
||||||
|
<name of job>` to the PR description. All such specified jobs will be executed
|
||||||
|
in the try build once the `@bors try` command is used on the PR. If no try
|
||||||
|
jobs are specified in this way, the jobs defined in the `try` section of
|
||||||
|
[`jobs.yml`] will be executed by default.
|
||||||
|
|
||||||
Try jobs are defined in the `try` section of [`jobs.yml`]. They are executed on the `try` branch under the `rust-lang-ci/rust` repository[^rust-lang-ci] and their results can be seen [here](https://github.com/rust-lang-ci/rust/actions), although usually you will be notified of the result by a comment made by bors on the corresponding PR.
|
> **Using `try-job` PR description directives**
|
||||||
|
>
|
||||||
|
> 1. Identify which set of try-jobs (max 10) you would like to exercise. You can
|
||||||
|
> find the name of the CI jobs in [`jobs.yml`].
|
||||||
|
>
|
||||||
|
> 2. Amend PR description to include (usually at the end of the PR description)
|
||||||
|
> e.g.
|
||||||
|
>
|
||||||
|
> ```text
|
||||||
|
> This PR fixes #123456.
|
||||||
|
>
|
||||||
|
> try-job: x86_64-msvc
|
||||||
|
> try-job: test-various
|
||||||
|
> ```
|
||||||
|
>
|
||||||
|
> Each `try-job` directive must be on its own line.
|
||||||
|
>
|
||||||
|
> 3. Run the prescribed try jobs with `@bors try`. As aforementioned, this
|
||||||
|
> requires the user to either (1) have `try` permissions or (2) be delegated
|
||||||
|
> with `try` permissions by `@bors delegate` by someone who has `try`
|
||||||
|
> permissions.
|
||||||
|
>
|
||||||
|
> Note that this is usually easier to do than manually edit [`jobs.yml`].
|
||||||
|
> However, it can be less flexible because you cannot adjust the set of tests
|
||||||
|
> that are exercised this way.
|
||||||
|
|
||||||
Multiple try builds can execute concurrently across different PRs, but inside a single PR, at most one try build can be executing at the same time.
|
Try jobs are defined in the `try` section of [`jobs.yml`]. They are executed on
|
||||||
|
the `try` branch under the `rust-lang-ci/rust` repository[^rust-lang-ci] and
|
||||||
|
their results can be seen [here](https://github.com/rust-lang-ci/rust/actions),
|
||||||
|
although usually you will be notified of the result by a comment made by bors on
|
||||||
|
the corresponding PR.
|
||||||
|
|
||||||
|
Multiple try builds can execute concurrently across different PRs.
|
||||||
|
|
||||||
|
<div class="warning">
|
||||||
|
bors identify try jobs by commit hash. This means that if you have two PRs
|
||||||
|
containing the same (latest) commits, running `@bors try` will result in the
|
||||||
|
*same* try job and it really confuses `bors`. Please refrain from doing so.
|
||||||
|
</div>
|
||||||
|
|
||||||
[rustc-perf]: https://github.com/rust-lang/rustc-perf
|
[rustc-perf]: https://github.com/rust-lang/rustc-perf
|
||||||
[crater]: https://github.com/rust-lang/crater
|
[crater]: https://github.com/rust-lang/crater
|
||||||
|
|
||||||
### Modifying CI jobs
|
### Modifying CI jobs
|
||||||
|
|
||||||
If you want to modify what gets executed on our CI, you can simply modify the `pr`, `auto` or `try` sections of the [`jobs.yml`] file.
|
If you want to modify what gets executed on our CI, you can simply modify the
|
||||||
|
`pr`, `auto` or `try` sections of the [`jobs.yml`] file.
|
||||||
|
|
||||||
You can also modify what gets executed temporarily, for example to test a particular platform
|
You can also modify what gets executed temporarily, for example to test a
|
||||||
or configuration that is challenging to test locally (for example, if a Windows build fails, but you don't have access to a Windows machine). Don't hesitate to use CI resources in such situations to try out a fix!
|
particular platform or configuration that is challenging to test locally (for
|
||||||
|
example, if a Windows build fails, but you don't have access to a Windows
|
||||||
|
machine). Don't hesitate to use CI resources in such situations to try out a
|
||||||
|
fix!
|
||||||
|
|
||||||
You can perform an arbitrary CI job in two ways:
|
You can perform an arbitrary CI job in two ways:
|
||||||
- Use the [try build](#try-builds) functionality, and specify the CI jobs that you want to be
|
- Use the [try build](#try-builds) functionality, and specify the CI jobs that
|
||||||
executed in try builds in your PR description.
|
you want to be executed in try builds in your PR description.
|
||||||
- Modify the [`pr`](#pull-request-builds) section of `jobs.yml` to specify which CI jobs should be
|
- Modify the [`pr`](#pull-request-builds) section of `jobs.yml` to specify which
|
||||||
executed after each push to your PR. This might be faster than repeatedly starting try builds.
|
CI jobs should be executed after each push to your PR. This might be faster
|
||||||
|
than repeatedly starting try builds.
|
||||||
|
|
||||||
To modify the jobs executed after each push to a PR, you can simply copy one of the job definitions from the `auto` section to the `pr` section. For example, the `x86_64-msvc` job is responsible for running the 64-bit MSVC tests.
|
To modify the jobs executed after each push to a PR, you can simply copy one of
|
||||||
You can copy it to the `pr` section to cause it to be executed after a commit is pushed to your
|
the job definitions from the `auto` section to the `pr` section. For example,
|
||||||
PR, like this:
|
the `x86_64-msvc` job is responsible for running the 64-bit MSVC tests. You can
|
||||||
|
copy it to the `pr` section to cause it to be executed after a commit is pushed
|
||||||
|
to your PR, like this:
|
||||||
|
|
||||||
```yaml
|
```yaml
|
||||||
pr:
|
pr:
|
||||||
|
|
@ -109,61 +221,69 @@ pr:
|
||||||
<<: *job-windows-8c
|
<<: *job-windows-8c
|
||||||
```
|
```
|
||||||
|
|
||||||
Then you can commit the file and push it to your PR branch on GitHub. GitHub Actions should then
|
Then you can commit the file and push it to your PR branch on GitHub. GitHub
|
||||||
execute this CI job after each push to your PR.
|
Actions should then execute this CI job after each push to your PR.
|
||||||
|
|
||||||
**After you have finished your experiments, don't forget to remove any changes you have made to `jobs.yml`, if they were supposed to be temporary!**
|
<div class="warning">
|
||||||
|
|
||||||
|
**After you have finished your experiments, don't forget to remove any changes
|
||||||
|
you have made to `jobs.yml`, if they were supposed to be temporary!**
|
||||||
|
|
||||||
|
A good practice is to prefix `[WIP]` in PR title while still running try jobs
|
||||||
|
and `[DO NOT MERGE]` in the commit that modifies the CI jobs for testing
|
||||||
|
purposes.
|
||||||
|
</div>
|
||||||
|
|
||||||
Although you are welcome to use CI, just be conscious that this is a shared
|
Although you are welcome to use CI, just be conscious that this is a shared
|
||||||
resource with limited concurrency. Try not to enable too many jobs at once (one or two should be sufficient in
|
resource with limited concurrency. Try not to enable too many jobs at once (one
|
||||||
most cases).
|
or two should be sufficient in most cases).
|
||||||
|
|
||||||
## Merging PRs serially with bors
|
## Merging PRs serially with bors
|
||||||
|
|
||||||
CI services usually test the last commit of a branch merged with the last
|
CI services usually test the last commit of a branch merged with the last commit
|
||||||
commit in `master`, and while that’s great to check if the feature works in
|
in `master`, and while that’s great to check if the feature works in isolation,
|
||||||
isolation, it doesn’t provide any guarantee the code is going to work once it’s
|
it doesn’t provide any guarantee the code is going to work once it’s merged.
|
||||||
merged. Breakages like these usually happen when another, incompatible PR is
|
Breakages like these usually happen when another, incompatible PR is merged
|
||||||
merged after the build happened.
|
after the build happened.
|
||||||
|
|
||||||
To ensure a `master` branch that works all the time, we forbid manual merges. Instead,
|
To ensure a `master` branch that works all the time, we forbid manual merges.
|
||||||
all PRs have to be approved through our bot, [bors] (the software behind it is
|
Instead, all PRs have to be approved through our bot, [bors] (the software
|
||||||
called [homu]). All the approved PRs are put [in a queue][merge queue] (sorted
|
behind it is called [homu]). All the approved PRs are put in a [merge queue]
|
||||||
by priority and creation date) and are automatically tested one at the time. If
|
(sorted by priority and creation date) and are automatically tested one at the
|
||||||
all the builders are green, the PR is merged, otherwise the failure is recorded
|
time. If all the builders are green, the PR is merged, otherwise the failure is
|
||||||
and the PR will have to be re-approved again.
|
recorded and the PR will have to be re-approved again.
|
||||||
|
|
||||||
Bors doesn’t interact with CI services directly, but it works by pushing the
|
Bors doesn’t interact with CI services directly, but it works by pushing the
|
||||||
merge commit it wants to test to specific branches (like `auto` or `try`), which
|
merge commit it wants to test to specific branches (like `auto` or `try`), which
|
||||||
are configured to execute CI checks. Bors then detects the
|
are configured to execute CI checks. Bors then detects the outcome of the build
|
||||||
outcome of the build by listening for either Commit Statuses or Check Runs.
|
by listening for either Commit Statuses or Check Runs. Since the merge commit is
|
||||||
Since the merge commit is based on the latest `master` and only one can be tested
|
based on the latest `master` and only one can be tested at the same time, when
|
||||||
at the same time, when the results are green, `master` is fast-forwarded to that
|
the results are green, `master` is fast-forwarded to that merge commit.
|
||||||
merge commit.
|
|
||||||
|
|
||||||
Unfortunately testing a single PR at the time, combined with our long CI (~2
|
Unfortunately testing a single PR at the time, combined with our long CI (~2
|
||||||
hours for a full run), means we can’t merge too many PRs in a single day, and a
|
hours for a full run), means we can’t merge too many PRs in a single day, and a
|
||||||
single failure greatly impacts our throughput for the day. The maximum number
|
single failure greatly impacts our throughput for the day. The maximum number of
|
||||||
of PRs we can merge in a day is around ~10.
|
PRs we can merge in a day is around ~10.
|
||||||
|
|
||||||
The large CI run times and requirement for a large builder pool is largely due to the
|
The large CI run times and requirement for a large builder pool is largely due
|
||||||
fact that full release artifacts are built in the `dist-` builders. This is worth it
|
to the fact that full release artifacts are built in the `dist-` builders. This
|
||||||
because these release artifacts:
|
is worth it because these release artifacts:
|
||||||
|
|
||||||
- Allow perf testing even at a later date
|
- Allow perf testing even at a later date.
|
||||||
- Allow bisection when bugs are discovered later
|
- Allow bisection when bugs are discovered later.
|
||||||
- Ensure release quality since if we're always releasing, we can catch problems early
|
- Ensure release quality since if we're always releasing, we can catch problems
|
||||||
|
early.
|
||||||
|
|
||||||
### Rollups
|
### Rollups
|
||||||
|
|
||||||
Some PRs don’t need the full test suite to be executed: trivial changes like
|
Some PRs don’t need the full test suite to be executed: trivial changes like
|
||||||
typo fixes or README improvements *shouldn’t* break the build, and testing
|
typo fixes or README improvements *shouldn’t* break the build, and testing every
|
||||||
every single one of them for 2+ hours is a big waste of time. To solve this,
|
single one of them for 2+ hours is a big waste of time. To solve this, we
|
||||||
we regularly create a "rollup", a PR where we merge several pending trivial
|
regularly create a "rollup", a PR where we merge several pending trivial PRs so
|
||||||
PRs so they can be tested together. Rollups are created manually by a team member
|
they can be tested together. Rollups are created manually by a team member using
|
||||||
using the "create a rollup" button on the [merge queue]. The team member uses their
|
the "create a rollup" button on the [merge queue]. The team member uses their
|
||||||
judgment to decide if a PR is risky or not, and are the best tool we have at
|
judgment to decide if a PR is risky or not, and are the best tool we have at the
|
||||||
the moment to keep the queue in a manageable state.
|
moment to keep the queue in a manageable state.
|
||||||
|
|
||||||
## Docker
|
## Docker
|
||||||
|
|
||||||
|
|
@ -174,20 +294,25 @@ platform’s custom [Docker container]. This has a lot of advantages for us:
|
||||||
underlying image (switching from the trusty image to xenial was painless for
|
underlying image (switching from the trusty image to xenial was painless for
|
||||||
us).
|
us).
|
||||||
- We can use ancient build environments to ensure maximum binary compatibility,
|
- We can use ancient build environments to ensure maximum binary compatibility,
|
||||||
for example [using older CentOS releases][dist-x86_64-linux] on our Linux builders.
|
for example [using older CentOS releases][dist-x86_64-linux] on our Linux
|
||||||
- We can avoid reinstalling tools (like QEMU or the Android emulator) every
|
builders.
|
||||||
time thanks to Docker image caching.
|
- We can avoid reinstalling tools (like QEMU or the Android emulator) every time
|
||||||
|
thanks to Docker image caching.
|
||||||
- Users can run the same tests in the same environment locally by just running
|
- Users can run the same tests in the same environment locally by just running
|
||||||
`src/ci/docker/run.sh image-name`, which is awesome to debug failures.
|
`src/ci/docker/run.sh image-name`, which is awesome to debug failures. Note
|
||||||
|
that there are only linux docker images available locally due to licensing and
|
||||||
|
other restrictions.
|
||||||
|
|
||||||
The docker images prefixed with `dist-` are used for building artifacts while those without that prefix run tests and checks.
|
The docker images prefixed with `dist-` are used for building artifacts while
|
||||||
|
those without that prefix run tests and checks.
|
||||||
|
|
||||||
We also run tests for less common architectures (mainly Tier 2 and Tier 3
|
We also run tests for less common architectures (mainly Tier 2 and Tier 3
|
||||||
platforms) in CI. Since those platforms are not x86 we either run
|
platforms) in CI. Since those platforms are not x86 we either run everything
|
||||||
everything inside QEMU or just cross-compile if we don’t want to run the tests
|
inside QEMU or just cross-compile if we don’t want to run the tests for that
|
||||||
for that platform.
|
platform.
|
||||||
|
|
||||||
These builders are running on a special pool of builders set up and maintained for us by GitHub.
|
These builders are running on a special pool of builders set up and maintained
|
||||||
|
for us by GitHub.
|
||||||
|
|
||||||
[Docker container]: https://github.com/rust-lang/rust/tree/master/src/ci/docker
|
[Docker container]: https://github.com/rust-lang/rust/tree/master/src/ci/docker
|
||||||
|
|
||||||
|
|
@ -198,10 +323,11 @@ Our CI workflow uses various caching mechanisms, mainly for two things:
|
||||||
### Docker images caching
|
### Docker images caching
|
||||||
|
|
||||||
The Docker images we use to run most of the Linux-based builders take a *long*
|
The Docker images we use to run most of the Linux-based builders take a *long*
|
||||||
time to fully build. To speed up the build, we cache it using [Docker registry caching],
|
time to fully build. To speed up the build, we cache it using [Docker registry
|
||||||
with the intermediate artifacts being stored on [ghcr.io]. We also push the built
|
caching], with the intermediate artifacts being stored on [ghcr.io]. We also
|
||||||
Docker images to ghcr, so that they can be reused by other tools (rustup) or
|
push the built Docker images to ghcr, so that they can be reused by other tools
|
||||||
by developers running the Docker build locally (to speed up their build).
|
(rustup) or by developers running the Docker build locally (to speed up their
|
||||||
|
build).
|
||||||
|
|
||||||
Since we test multiple, diverged branches (`master`, `beta` and `stable`), we
|
Since we test multiple, diverged branches (`master`, `beta` and `stable`), we
|
||||||
can’t rely on a single cache for the images, otherwise builds on a branch would
|
can’t rely on a single cache for the images, otherwise builds on a branch would
|
||||||
|
|
@ -216,8 +342,9 @@ Dockerfiles and related scripts.
|
||||||
|
|
||||||
We build some C/C++ stuff in various CI jobs, and we rely on [sccache] to cache
|
We build some C/C++ stuff in various CI jobs, and we rely on [sccache] to cache
|
||||||
the intermediate LLVM artifacts. Sccache is a distributed ccache developed by
|
the intermediate LLVM artifacts. Sccache is a distributed ccache developed by
|
||||||
Mozilla, which can use an object storage bucket as the storage backend. In our case,
|
Mozilla, which can use an object storage bucket as the storage backend. In our
|
||||||
the artefacts are uploaded to an S3 bucket that we control (`rust-lang-ci-sccache2`).
|
case, the artefacts are uploaded to an S3 bucket that we control
|
||||||
|
(`rust-lang-ci-sccache2`).
|
||||||
|
|
||||||
[sccache]: https://github.com/mozilla/sccache
|
[sccache]: https://github.com/mozilla/sccache
|
||||||
|
|
||||||
|
|
@ -228,16 +355,16 @@ During the years we developed some custom tooling to improve our CI experience.
|
||||||
### Rust Log Analyzer to show the error message in PRs
|
### Rust Log Analyzer to show the error message in PRs
|
||||||
|
|
||||||
The build logs for `rust-lang/rust` are huge, and it’s not practical to find
|
The build logs for `rust-lang/rust` are huge, and it’s not practical to find
|
||||||
what caused the build to fail by looking at the logs. To improve the
|
what caused the build to fail by looking at the logs. To improve the developers’
|
||||||
developers’ experience we developed a bot called [Rust Log Analyzer][rla] (RLA)
|
experience we developed a bot called [Rust Log Analyzer][rla] (RLA) that
|
||||||
that receives the build logs on failure and extracts the error message
|
receives the build logs on failure and extracts the error message automatically,
|
||||||
automatically, posting it on the PR.
|
posting it on the PR.
|
||||||
|
|
||||||
The bot is not hardcoded to look for error strings, but was trained with a
|
The bot is not hardcoded to look for error strings, but was trained with a bunch
|
||||||
bunch of build failures to recognize which lines are common between builds and
|
of build failures to recognize which lines are common between builds and which
|
||||||
which are not. While the generated snippets can be weird sometimes, the bot is
|
are not. While the generated snippets can be weird sometimes, the bot is pretty
|
||||||
pretty good at identifying the relevant lines even if it’s an error we've never
|
good at identifying the relevant lines even if it’s an error we've never seen
|
||||||
seen before.
|
before.
|
||||||
|
|
||||||
[rla]: https://github.com/rust-lang/rust-log-analyzer
|
[rla]: https://github.com/rust-lang/rust-log-analyzer
|
||||||
|
|
||||||
|
|
@ -245,16 +372,16 @@ seen before.
|
||||||
|
|
||||||
The `rust-lang/rust` repo doesn’t only test the compiler on its CI, but also a
|
The `rust-lang/rust` repo doesn’t only test the compiler on its CI, but also a
|
||||||
variety of tools and documentation. Some documentation is pulled in via git
|
variety of tools and documentation. Some documentation is pulled in via git
|
||||||
submodules. If we blocked merging rustc PRs on the documentation being fixed,
|
submodules. If we blocked merging rustc PRs on the documentation being fixed, we
|
||||||
we would be stuck in a chicken-and-egg problem, because the documentation's CI
|
would be stuck in a chicken-and-egg problem, because the documentation's CI
|
||||||
would not pass since updating it would need the not-yet-merged version of
|
would not pass since updating it would need the not-yet-merged version of rustc
|
||||||
rustc to test against (and we usually require CI to be passing).
|
to test against (and we usually require CI to be passing).
|
||||||
|
|
||||||
To avoid the problem, submodules are allowed to fail, and their status is
|
To avoid the problem, submodules are allowed to fail, and their status is
|
||||||
recorded in [rust-toolstate]. When a submodule breaks, a bot automatically
|
recorded in [rust-toolstate]. When a submodule breaks, a bot automatically pings
|
||||||
pings the maintainers so they know about the breakage, and it records the
|
the maintainers so they know about the breakage, and it records the failure on
|
||||||
failure on the toolstate repository. The release process will then ignore
|
the toolstate repository. The release process will then ignore broken tools on
|
||||||
broken tools on nightly, removing them from the shipped nightlies.
|
nightly, removing them from the shipped nightlies.
|
||||||
|
|
||||||
While tool failures are allowed most of the time, they’re automatically
|
While tool failures are allowed most of the time, they’re automatically
|
||||||
forbidden a week before a release: we don’t care if tools are broken on nightly
|
forbidden a week before a release: we don’t care if tools are broken on nightly
|
||||||
|
|
|
||||||
|
|
@ -4,158 +4,163 @@
|
||||||
|
|
||||||
## Introduction
|
## Introduction
|
||||||
|
|
||||||
`compiletest` is the main test harness of the Rust test suite.
|
`compiletest` is the main test harness of the Rust test suite. It allows test
|
||||||
It allows test authors to organize large numbers of tests
|
authors to organize large numbers of tests (the Rust compiler has many
|
||||||
(the Rust compiler has many thousands),
|
thousands), efficient test execution (parallel execution is supported), and
|
||||||
efficient test execution (parallel execution is supported),
|
allows the test author to configure behavior and expected results of both
|
||||||
and allows the test author to configure behavior and expected results of both
|
|
||||||
individual and groups of tests.
|
individual and groups of tests.
|
||||||
|
|
||||||
> NOTE:
|
> **Note for macOS users**
|
||||||
> For macOS users, `SIP` (System Integrity Protection) [may consistently
|
|
||||||
> check the compiled binary by sending network requests to Apple][zulip],
|
|
||||||
> so you may get a huge performance degradation when running tests.
|
|
||||||
>
|
>
|
||||||
> You can resolve it by tweaking the following settings:
|
> For macOS users, `SIP` (System Integrity Protection) [may consistently check
|
||||||
> `Privacy & Security -> Developer Tools -> Add Terminal (Or VsCode, etc.)`.
|
> the compiled binary by sending network requests to Apple][zulip], so you may
|
||||||
|
> get a huge performance degradation when running tests.
|
||||||
|
>
|
||||||
|
> You can resolve it by tweaking the following settings: `Privacy & Security ->
|
||||||
|
> Developer Tools -> Add Terminal (Or VsCode, etc.)`.
|
||||||
|
|
||||||
[zulip]: https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/.E2.9C.94.20Is.20there.20any.20performance.20issue.20for.20MacOS.3F
|
[zulip]: https://rust-lang.zulipchat.com/#narrow/stream/182449-t-compiler.2Fhelp/topic/.E2.9C.94.20Is.20there.20any.20performance.20issue.20for.20MacOS.3F
|
||||||
|
|
||||||
`compiletest` may check test code for success, for runtime failure,
|
`compiletest` may check test code for compile-time or run-time success/failure.
|
||||||
or for compile-time failure.
|
|
||||||
Tests are typically organized as a Rust source file with annotations in
|
|
||||||
comments before and/or within the test code.
|
|
||||||
These comments serve to direct `compiletest` on if or how to run the test,
|
|
||||||
what behavior to expect, and more.
|
|
||||||
See [header commands](headers.md) and the test suite documentation below
|
|
||||||
for more details on these annotations.
|
|
||||||
|
|
||||||
See the [Adding new tests](adding.md) chapter for a tutorial on creating a new
|
Tests are typically organized as a Rust source file with annotations in comments
|
||||||
test, and the [Running tests](running.md) chapter on how to run the test
|
before and/or within the test code. These comments serve to direct `compiletest`
|
||||||
suite.
|
on if or how to run the test, what behavior to expect, and more. See
|
||||||
|
[directives](directives.md) and the test suite documentation below for more details
|
||||||
|
on these annotations.
|
||||||
|
|
||||||
Compiletest itself tries to avoid running tests when the artifacts
|
See the [Adding new tests](adding.md) and [Best practies](best-practiecs.md)
|
||||||
that are involved (mainly the compiler) haven't changed. You can use
|
chapters for a tutorial on creating a new test and advice on writing a good
|
||||||
`x test --test-args --force-rerun` to rerun a test even when none of the
|
test, and the [Running tests](running.md) chapter on how to run the test suite.
|
||||||
inputs have changed.
|
|
||||||
|
Compiletest itself tries to avoid running tests when the artifacts that are
|
||||||
|
involved (mainly the compiler) haven't changed. You can use `x test --test-args
|
||||||
|
--force-rerun` to rerun a test even when none of the inputs have changed.
|
||||||
|
|
||||||
## Test suites
|
## Test suites
|
||||||
|
|
||||||
All of the tests are in the [`tests`] directory.
|
All of the tests are in the [`tests`] directory. The tests are organized into
|
||||||
The tests are organized into "suites", with each suite in a separate subdirectory.
|
"suites", with each suite in a separate subdirectory. Each test suite behaves a
|
||||||
Each test suite behaves a little differently, with different compiler behavior
|
little differently, with different compiler behavior and different checks for
|
||||||
and different checks for correctness.
|
correctness. For example, the [`tests/incremental`] directory contains tests for
|
||||||
For example, the [`tests/incremental`] directory contains tests for
|
incremental compilation. The various suites are defined in
|
||||||
incremental compilation.
|
[`src/tools/compiletest/src/common.rs`] in the `pub enum Mode` declaration.
|
||||||
The various suites are defined in [`src/tools/compiletest/src/common.rs`] in
|
|
||||||
the `pub enum Mode` declaration.
|
|
||||||
|
|
||||||
The following test suites are available, with links for more information:
|
The following test suites are available, with links for more information:
|
||||||
|
|
||||||
- [`ui`](ui.md) — tests that check the stdout/stderr from the compilation
|
### Compiler-specific test suites
|
||||||
and/or running the resulting executable
|
|
||||||
- `ui-fulldeps` — `ui` tests which require a linkable build of `rustc` (such
|
| Test suite | Purpose |
|
||||||
as using `extern crate rustc_span;` or used as a plugin)
|
|-------------------------------------------|---------------------------------------------------------------------------------------------------------------------|
|
||||||
- [`pretty`](#pretty-printer-tests) — tests for pretty printing
|
| [`ui`](ui.md) | Check the stdout/stderr snapshots from the compilation and/or running the resulting executable |
|
||||||
- [`incremental`](#incremental-tests) — tests incremental compilation behavior
|
| `ui-fulldeps` | `ui` tests which require a linkable build of `rustc` (such as using `extern crate rustc_span;` or used as a plugin) |
|
||||||
- [`debuginfo`](#debuginfo-tests) — tests for debuginfo generation running debuggers
|
| [`pretty`](#pretty-printer-tests) | Check pretty printing |
|
||||||
- [`codegen`](#codegen-tests) — tests for code generation
|
| [`incremental`](#incremental-tests) | Check incremental compilation behavior |
|
||||||
- [`codegen-units`](#codegen-units-tests) — tests for codegen unit partitioning
|
| [`debuginfo`](#debuginfo-tests) | Check debuginfo generation running debuggers |
|
||||||
- [`assembly`](#assembly-tests) — verifies assembly output
|
| [`codegen`](#codegen-tests) | Check code generation |
|
||||||
- [`mir-opt`](#mir-opt-tests) — tests for MIR generation
|
| [`codegen-units`](#codegen-units-tests) | Check codegen unit partitioning |
|
||||||
- [`run-make`](#run-make-tests) — general purpose tests using Rust programs (or
|
| [`assembly`](#assembly-tests) | Check assembly output |
|
||||||
Makefiles (legacy))
|
| [`mir-opt`](#mir-opt-tests) | Check MIR generation and optimizations |
|
||||||
- [`run-pass-valgrind`](#valgrind-tests) — tests run with Valgrind
|
| [`run-pass-valgrind`](#valgrind-tests) | Run with Valgrind |
|
||||||
- [`coverage`](#coverage-tests) - tests for coverage instrumentation
|
| [`coverage`](#coverage-tests) | Check coverage instrumentation |
|
||||||
- [`coverage-run-rustdoc`](#coverage-tests) - coverage tests that also run
|
| [`coverage-run-rustdoc`](#coverage-tests) | `coverage` tests that also run instrumented doctests |
|
||||||
instrumented doctests
|
|
||||||
- [Rustdoc tests](../rustdoc.md#tests):
|
### General purpose test suite
|
||||||
- `rustdoc` — tests for rustdoc, making sure that the generated files
|
|
||||||
contain the expected documentation.
|
[`run-make`](#run-make-tests) are general purpose tests using Rust programs (or
|
||||||
- `rustdoc-gui` — tests for rustdoc's GUI using a web browser.
|
Makefiles (legacy)).
|
||||||
- `rustdoc-js` — tests to ensure the rustdoc search is working as expected.
|
|
||||||
- `rustdoc-js-std` — tests to ensure the rustdoc search is working as expected
|
### Rustdoc test suites
|
||||||
(run specifically on the std docs).
|
|
||||||
- `rustdoc-json` — tests on the JSON output of rustdoc.
|
See [Rustdoc tests](../rustdoc.md#tests) for more details.
|
||||||
- `rustdoc-ui` — tests on the terminal output of rustdoc.
|
|
||||||
|
| Test suite | Purpose |
|
||||||
|
|------------------|--------------------------------------------------------------------------|
|
||||||
|
| `rustdoc` | Check `rustdoc` generated files contain the expected documentation |
|
||||||
|
| `rustdoc-gui` | Check `rustdoc`'s GUI using a web browser |
|
||||||
|
| `rustdoc-js` | Check `rustdoc` search is working as expected |
|
||||||
|
| `rustdoc-js-std` | Check rustdoc search is working as expected specifically on the std docs |
|
||||||
|
| `rustdoc-json` | Check JSON output of `rustdoc` |
|
||||||
|
| `rustdoc-ui` | Check terminal output of `rustdoc` |
|
||||||
|
|
||||||
[`tests`]: https://github.com/rust-lang/rust/blob/master/tests
|
[`tests`]: https://github.com/rust-lang/rust/blob/master/tests
|
||||||
[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
|
[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
|
||||||
|
|
||||||
### Pretty-printer tests
|
### Pretty-printer tests
|
||||||
|
|
||||||
The tests in [`tests/pretty`] exercise the "pretty-printing" functionality of `rustc`.
|
The tests in [`tests/pretty`] exercise the "pretty-printing" functionality of
|
||||||
The `-Z unpretty` CLI option for `rustc` causes it to translate the input source
|
`rustc`. The `-Z unpretty` CLI option for `rustc` causes it to translate the
|
||||||
into various different formats, such as the Rust source after macro expansion.
|
input source into various different formats, such as the Rust source after macro
|
||||||
|
expansion.
|
||||||
|
|
||||||
The pretty-printer tests have several [header commands](headers.md) described below.
|
The pretty-printer tests have several [directives](directives.md) described below.
|
||||||
These commands can significantly change the behavior of the test, but the
|
These commands can significantly change the behavior of the test, but the
|
||||||
default behavior without any commands is to:
|
default behavior without any commands is to:
|
||||||
|
|
||||||
1. Run `rustc -Zunpretty=normal` on the source file
|
1. Run `rustc -Zunpretty=normal` on the source file.
|
||||||
2. Run `rustc -Zunpretty=normal` on the output of the previous step
|
2. Run `rustc -Zunpretty=normal` on the output of the previous step.
|
||||||
3. The output of the previous two steps should be the same.
|
3. The output of the previous two steps should be the same.
|
||||||
4. Run `rustc -Zno-codegen` on the output to make sure that it can type check
|
4. Run `rustc -Zno-codegen` on the output to make sure that it can type check
|
||||||
(this is similar to running `cargo check`)
|
(this is similar to running `cargo check`).
|
||||||
|
|
||||||
If any of the commands above fail, then the test fails.
|
If any of the commands above fail, then the test fails.
|
||||||
|
|
||||||
The header commands for pretty-printing tests are:
|
The directives for pretty-printing tests are:
|
||||||
|
|
||||||
* `pretty-mode` specifies the mode pretty-print tests should run in
|
- `pretty-mode` specifies the mode pretty-print tests should run in (that is,
|
||||||
(that is, the argument to `-Zunpretty`).
|
the argument to `-Zunpretty`). The default is `normal` if not specified.
|
||||||
The default is `normal` if not specified.
|
- `pretty-compare-only` causes a pretty test to only compare the pretty-printed
|
||||||
* `pretty-compare-only` causes a pretty test to only compare the pretty-printed output
|
output (stopping after step 3 from above). It will not try to compile the
|
||||||
(stopping after step 3 from above).
|
expanded output to type check it. This is needed for a pretty-mode that does
|
||||||
It will not try to compile the expanded output to type check it.
|
not expand to valid Rust, or for other situations where the expanded output
|
||||||
This is needed for a pretty-mode that does not expand to valid
|
cannot be compiled.
|
||||||
Rust, or for other situations where the expanded output cannot be compiled.
|
- `pretty-expanded` allows a pretty test to also check that the expanded output
|
||||||
* `pretty-expanded` allows a pretty test to also check that the expanded
|
can be type checked. That is, after the steps above, it does two more steps:
|
||||||
output can be type checked.
|
|
||||||
That is, after the steps above, it does two more steps:
|
|
||||||
|
|
||||||
> 5. Run `rustc -Zunpretty=expanded` on the original source
|
> 5. Run `rustc -Zunpretty=expanded` on the original source
|
||||||
> 6. Run `rustc -Zno-codegen` on the expanded output to make sure that it can type check
|
> 6. Run `rustc -Zno-codegen` on the expanded output to make sure that it can type check
|
||||||
|
|
||||||
This is needed because not all code can be compiled after being expanded.
|
This is needed because not all code can be compiled after being expanded.
|
||||||
Pretty tests should specify this if they can.
|
Pretty tests should specify this if they can. An example where this cannot be
|
||||||
An example where this cannot be used is if the test includes `println!`.
|
used is if the test includes `println!`. That macro expands to reference
|
||||||
That macro expands to reference private internal functions of the standard
|
private internal functions of the standard library that cannot be called
|
||||||
library that cannot be called directly without the `fmt_internals` feature
|
directly without the `fmt_internals` feature gate.
|
||||||
gate.
|
|
||||||
|
|
||||||
More history about this may be found in
|
More history about this may be found in
|
||||||
[#23616](https://github.com/rust-lang/rust/issues/23616#issuecomment-484999901).
|
[#23616](https://github.com/rust-lang/rust/issues/23616#issuecomment-484999901).
|
||||||
* `pp-exact` is used to ensure a pretty-print test results in specific output.
|
- `pp-exact` is used to ensure a pretty-print test results in specific output.
|
||||||
If specified without a value, then it means the pretty-print output should
|
If specified without a value, then it means the pretty-print output should
|
||||||
match the original source.
|
match the original source. If specified with a value, as in `//@
|
||||||
If specified with a value, as in `// pp-exact:foo.pp`,
|
pp-exact:foo.pp`, it will ensure that the pretty-printed output matches the
|
||||||
it will ensure that the pretty-printed output matches the contents of the given file.
|
contents of the given file. Otherwise, if `pp-exact` is not specified, then
|
||||||
Otherwise, if `pp-exact` is not specified, then the pretty-printed output
|
the pretty-printed output will be pretty-printed one more time, and the output
|
||||||
will be pretty-printed one more time, and the output of the two
|
of the two pretty-printing rounds will be compared to ensure that the
|
||||||
pretty-printing rounds will be compared to ensure that the pretty-printed
|
pretty-printed output converges to a steady state.
|
||||||
output converges to a steady state.
|
|
||||||
|
|
||||||
[`tests/pretty`]: https://github.com/rust-lang/rust/tree/master/tests/pretty
|
[`tests/pretty`]: https://github.com/rust-lang/rust/tree/master/tests/pretty
|
||||||
|
|
||||||
### Incremental tests
|
### Incremental tests
|
||||||
|
|
||||||
The tests in [`tests/incremental`] exercise incremental compilation.
|
The tests in [`tests/incremental`] exercise incremental compilation. They use
|
||||||
They use [revision headers](#revisions) to tell compiletest to run the
|
[`revisions` directive](#revisions) to tell compiletest to run the compiler in a
|
||||||
compiler in a series of steps.
|
series of steps.
|
||||||
|
|
||||||
Compiletest starts with an empty directory with the `-C incremental` flag, and
|
Compiletest starts with an empty directory with the `-C incremental` flag, and
|
||||||
then runs the compiler for each revision, reusing the incremental results from
|
then runs the compiler for each revision, reusing the incremental results from
|
||||||
previous steps.
|
previous steps.
|
||||||
|
|
||||||
The revisions should start with:
|
The revisions should start with:
|
||||||
|
|
||||||
* `rpass` — the test should compile and run successfully
|
* `rpass` — the test should compile and run successfully
|
||||||
* `rfail` — the test should compile successfully, but the executable should fail to run
|
* `rfail` — the test should compile successfully, but the executable should fail to run
|
||||||
* `cfail` — the test should fail to compile
|
* `cfail` — the test should fail to compile
|
||||||
|
|
||||||
To make the revisions unique, you should add a suffix like `rpass1` and `rpass2`.
|
To make the revisions unique, you should add a suffix like `rpass1` and
|
||||||
|
`rpass2`.
|
||||||
|
|
||||||
|
To simulate changing the source, compiletest also passes a `--cfg` flag with the
|
||||||
|
current revision name.
|
||||||
|
|
||||||
To simulate changing the source, compiletest also passes a `--cfg` flag with
|
|
||||||
the current revision name.
|
|
||||||
For example, this will run twice, simulating changing a function:
|
For example, this will run twice, simulating changing a function:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
|
|
@ -174,28 +179,27 @@ fn foo() {
|
||||||
fn main() { foo(); }
|
fn main() { foo(); }
|
||||||
```
|
```
|
||||||
|
|
||||||
`cfail` tests support the `forbid-output` header to specify that a certain
|
`cfail` tests support the `forbid-output` directive to specify that a certain
|
||||||
substring must not appear anywhere in the compiler output.
|
substring must not appear anywhere in the compiler output. This can be useful to
|
||||||
This can be useful to ensure certain errors do not appear, but this can be
|
ensure certain errors do not appear, but this can be fragile as error messages
|
||||||
fragile as error messages change over time, and a test may no longer be
|
change over time, and a test may no longer be checking the right thing but will
|
||||||
checking the right thing but will still pass.
|
still pass.
|
||||||
|
|
||||||
`cfail` tests support the `should-ice` header to specify that a test should
|
`cfail` tests support the `should-ice` directive to specify that a test should
|
||||||
cause an Internal Compiler Error (ICE).
|
cause an Internal Compiler Error (ICE). This is a highly specialized directive
|
||||||
This is a highly specialized header to check that the incremental cache
|
to check that the incremental cache continues to work after an ICE.
|
||||||
continues to work after an ICE.
|
|
||||||
|
|
||||||
[`tests/incremental`]: https://github.com/rust-lang/rust/tree/master/tests/incremental
|
[`tests/incremental`]: https://github.com/rust-lang/rust/tree/master/tests/incremental
|
||||||
|
|
||||||
|
|
||||||
### Debuginfo tests
|
### Debuginfo tests
|
||||||
|
|
||||||
The tests in [`tests/debuginfo`] test debuginfo generation.
|
The tests in [`tests/debuginfo`] test debuginfo generation. They build a
|
||||||
They build a program, launch a debugger, and issue commands to the debugger.
|
program, launch a debugger, and issue commands to the debugger. A single test
|
||||||
A single test can work with cdb, gdb, and lldb.
|
can work with cdb, gdb, and lldb.
|
||||||
|
|
||||||
Most tests should have the `// compile-flags: -g` header or something similar
|
Most tests should have the `//@ compile-flags: -g` directive or something
|
||||||
to generate the appropriate debuginfo.
|
similar to generate the appropriate debuginfo.
|
||||||
|
|
||||||
To set a breakpoint on a line, add a `// #break` comment on the line.
|
To set a breakpoint on a line, add a `// #break` comment on the line.
|
||||||
|
|
||||||
|
|
@ -205,15 +209,16 @@ The debuginfo tests consist of a series of debugger commands along with
|
||||||
The commands are comments of the form `// $DEBUGGER-command:$COMMAND` where
|
The commands are comments of the form `// $DEBUGGER-command:$COMMAND` where
|
||||||
`$DEBUGGER` is the debugger being used and `$COMMAND` is the debugger command
|
`$DEBUGGER` is the debugger being used and `$COMMAND` is the debugger command
|
||||||
to execute.
|
to execute.
|
||||||
|
|
||||||
The debugger values can be:
|
The debugger values can be:
|
||||||
|
|
||||||
* `cdb`
|
- `cdb`
|
||||||
* `gdb`
|
- `gdb`
|
||||||
* `gdbg` — GDB without Rust support (versions older than 7.11)
|
- `gdbg` — GDB without Rust support (versions older than 7.11)
|
||||||
* `gdbr` — GDB with Rust support
|
- `gdbr` — GDB with Rust support
|
||||||
* `lldb`
|
- `lldb`
|
||||||
* `lldbg` — LLDB without Rust support
|
- `lldbg` — LLDB without Rust support
|
||||||
* `lldbr` — LLDB with Rust support (this no longer exists)
|
- `lldbr` — LLDB with Rust support (this no longer exists)
|
||||||
|
|
||||||
The command to check the output are of the form `// $DEBUGGER-check:$OUTPUT`
|
The command to check the output are of the form `// $DEBUGGER-check:$OUTPUT`
|
||||||
where `$OUTPUT` is the output to expect.
|
where `$OUTPUT` is the output to expect.
|
||||||
|
|
@ -237,30 +242,32 @@ fn main() {
|
||||||
fn b() {}
|
fn b() {}
|
||||||
```
|
```
|
||||||
|
|
||||||
The following [header commands](headers.md) are available to disable a
|
The following [directives](directives.md) are available to disable a test based on
|
||||||
test based on the debugger currently being used:
|
the debugger currently being used:
|
||||||
|
|
||||||
* `min-cdb-version: 10.0.18317.1001` — ignores the test if the version of cdb
|
- `min-cdb-version: 10.0.18317.1001` — ignores the test if the version of cdb
|
||||||
is below the given version
|
is below the given version
|
||||||
* `min-gdb-version: 8.2` — ignores the test if the version of gdb is below the
|
- `min-gdb-version: 8.2` — ignores the test if the version of gdb is below the
|
||||||
given version
|
given version
|
||||||
* `ignore-gdb-version: 9.2` — ignores the test if the version of gdb is equal
|
- `ignore-gdb-version: 9.2` — ignores the test if the version of gdb is equal
|
||||||
to the given version
|
to the given version
|
||||||
* `ignore-gdb-version: 7.11.90 - 8.0.9` — ignores the test if the version of
|
- `ignore-gdb-version: 7.11.90 - 8.0.9` — ignores the test if the version of
|
||||||
gdb is in a range (inclusive)
|
gdb is in a range (inclusive)
|
||||||
* `min-lldb-version: 310` — ignores the test if the version of lldb is below
|
- `min-lldb-version: 310` — ignores the test if the version of lldb is below
|
||||||
the given version
|
the given version
|
||||||
* `rust-lldb` — ignores the test if lldb is not contain the Rust plugin.
|
- `rust-lldb` — ignores the test if lldb is not contain the Rust plugin. NOTE:
|
||||||
NOTE: The "Rust" version of LLDB doesn't exist anymore, so this will always be ignored.
|
The "Rust" version of LLDB doesn't exist anymore, so this will always be
|
||||||
This should probably be removed.
|
ignored. This should probably be removed.
|
||||||
|
|
||||||
> **Note on running lldb debuginfo tests locally**
|
> **Note on running lldb debuginfo tests locally**
|
||||||
>
|
>
|
||||||
> If you want to run lldb debuginfo tests locally, then currently on Windows it is required that:
|
> If you want to run lldb debuginfo tests locally, then currently on Windows it
|
||||||
|
> is required that:
|
||||||
>
|
>
|
||||||
> - You have Python 3.10 installed.
|
> - You have Python 3.10 installed.
|
||||||
> - You have `python310.dll` available in your `PATH` env var. This is not provided by the standard
|
> - You have `python310.dll` available in your `PATH` env var. This is not
|
||||||
> Python installer you obtain from `python.org`; you need to add this to `PATH` manually.
|
> provided by the standard Python installer you obtain from `python.org`; you
|
||||||
|
> need to add this to `PATH` manually.
|
||||||
>
|
>
|
||||||
> Otherwise the lldb debuginfo tests can produce crashes in mysterious ways.
|
> Otherwise the lldb debuginfo tests can produce crashes in mysterious ways.
|
||||||
|
|
||||||
|
|
@ -269,11 +276,11 @@ test based on the debugger currently being used:
|
||||||
|
|
||||||
### Codegen tests
|
### Codegen tests
|
||||||
|
|
||||||
The tests in [`tests/codegen`] test LLVM code generation.
|
The tests in [`tests/codegen`] test LLVM code generation. They compile the test
|
||||||
They compile the test with the `--emit=llvm-ir` flag to emit LLVM IR.
|
with the `--emit=llvm-ir` flag to emit LLVM IR. They then run the LLVM
|
||||||
They then run the LLVM [FileCheck] tool.
|
[FileCheck] tool. The test is annotated with various `// CHECK` comments to
|
||||||
The test is annotated with various `// CHECK` comments to check the generated code.
|
check the generated code. See the [FileCheck] documentation for a tutorial and
|
||||||
See the FileCheck documentation for a tutorial and more information.
|
more information.
|
||||||
|
|
||||||
See also the [assembly tests](#assembly-tests) for a similar set of tests.
|
See also the [assembly tests](#assembly-tests) for a similar set of tests.
|
||||||
|
|
||||||
|
|
@ -283,18 +290,17 @@ See also the [assembly tests](#assembly-tests) for a similar set of tests.
|
||||||
|
|
||||||
### Assembly tests
|
### Assembly tests
|
||||||
|
|
||||||
The tests in [`tests/assembly`] test LLVM assembly output.
|
The tests in [`tests/assembly`] test LLVM assembly output. They compile the test
|
||||||
They compile the test with the `--emit=asm` flag to emit a `.s` file with the
|
with the `--emit=asm` flag to emit a `.s` file with the assembly output. They
|
||||||
assembly output.
|
then run the LLVM [FileCheck] tool.
|
||||||
They then run the LLVM [FileCheck] tool.
|
|
||||||
|
|
||||||
Each test should be annotated with the `// assembly-output:` header
|
Each test should be annotated with the `//@ assembly-output:` directive with a
|
||||||
with a value of either `emit-asm` or `ptx-linker` to indicate
|
value of either `emit-asm` or `ptx-linker` to indicate the type of assembly
|
||||||
the type of assembly output.
|
output.
|
||||||
|
|
||||||
Then, they should be annotated with various `// CHECK` comments to check the
|
Then, they should be annotated with various `// CHECK` comments to check the
|
||||||
assembly output.
|
assembly output. See the [FileCheck] documentation for a tutorial and more
|
||||||
See the FileCheck documentation for a tutorial and more information.
|
information.
|
||||||
|
|
||||||
See also the [codegen tests](#codegen-tests) for a similar set of tests.
|
See also the [codegen tests](#codegen-tests) for a similar set of tests.
|
||||||
|
|
||||||
|
|
@ -310,26 +316,27 @@ These tests work by running `rustc` with a flag to print the result of the
|
||||||
monomorphization collection pass, and then special annotations in the file are
|
monomorphization collection pass, and then special annotations in the file are
|
||||||
used to compare against that.
|
used to compare against that.
|
||||||
|
|
||||||
Each test should be annotated with the `// compile-flags:-Zprint-mono-items=VAL`
|
Each test should be annotated with the `//@
|
||||||
header with the appropriate VAL to instruct `rustc` to print the
|
compile-flags:-Zprint-mono-items=VAL` directive with the appropriate `VAL` to
|
||||||
monomorphization information.
|
instruct `rustc` to print the monomorphization information.
|
||||||
|
|
||||||
Then, the test should be annotated with comments of the form `//~ MONO_ITEM name`
|
Then, the test should be annotated with comments of the form `//~ MONO_ITEM
|
||||||
where `name` is the monomorphized string printed by rustc like `fn <u32 as Trait>::foo`.
|
name` where `name` is the monomorphized string printed by rustc like `fn <u32 as
|
||||||
|
Trait>::foo`.
|
||||||
|
|
||||||
To check for CGU partitioning, a comment of the form `//~ MONO_ITEM name @@ cgu`
|
To check for CGU partitioning, a comment of the form `//~ MONO_ITEM name @@ cgu`
|
||||||
where `cgu` is a space separated list of the CGU names and the linkage
|
where `cgu` is a space separated list of the CGU names and the linkage
|
||||||
information in brackets.
|
information in brackets. For example: `//~ MONO_ITEM static function::FOO @@
|
||||||
For example: `//~ MONO_ITEM static function::FOO @@ statics[Internal]`
|
statics[Internal]`
|
||||||
|
|
||||||
[`tests/codegen-units`]: https://github.com/rust-lang/rust/tree/master/tests/codegen-units
|
[`tests/codegen-units`]: https://github.com/rust-lang/rust/tree/master/tests/codegen-units
|
||||||
|
|
||||||
|
|
||||||
### Mir-opt tests
|
### Mir-opt tests
|
||||||
|
|
||||||
The tests in [`tests/mir-opt`] check parts of the generated MIR to make
|
The tests in [`tests/mir-opt`] check parts of the generated MIR to make sure it
|
||||||
sure it is generated correctly and is doing the expected optimizations.
|
is generated correctly and is doing the expected optimizations. Check out the
|
||||||
Check out the [MIR Optimizations](../mir/optimizations.md) chapter for more.
|
[MIR Optimizations](../mir/optimizations.md) chapter for more.
|
||||||
|
|
||||||
Compiletest will build the test with several flags to dump the MIR output and
|
Compiletest will build the test with several flags to dump the MIR output and
|
||||||
set a baseline for optimizations:
|
set a baseline for optimizations:
|
||||||
|
|
@ -341,29 +348,28 @@ set a baseline for optimizations:
|
||||||
* `-Zdump-mir-exclude-pass-number`
|
* `-Zdump-mir-exclude-pass-number`
|
||||||
|
|
||||||
The test should be annotated with `// EMIT_MIR` comments that specify files that
|
The test should be annotated with `// EMIT_MIR` comments that specify files that
|
||||||
will contain the expected MIR output.
|
will contain the expected MIR output. You can use `x test --bless` to create the
|
||||||
You can use `x test --bless` to create the initial expected files.
|
initial expected files.
|
||||||
|
|
||||||
There are several forms the `EMIT_MIR` comment can take:
|
There are several forms the `EMIT_MIR` comment can take:
|
||||||
|
|
||||||
* `// EMIT_MIR $MIR_PATH.mir` — This will check that the given filename
|
- `// EMIT_MIR $MIR_PATH.mir` — This will check that the given filename matches
|
||||||
matches the exact output from the MIR dump.
|
the exact output from the MIR dump. For example,
|
||||||
For example, `my_test.main.SimplifyCfg-elaborate-drops.after.mir` will load
|
`my_test.main.SimplifyCfg-elaborate-drops.after.mir` will load that file from
|
||||||
that file from the test directory, and compare it against the dump from
|
the test directory, and compare it against the dump from rustc.
|
||||||
rustc.
|
|
||||||
|
|
||||||
Checking the "after" file (which is after optimization) is useful if you are
|
Checking the "after" file (which is after optimization) is useful if you are
|
||||||
interested in the final state after an optimization.
|
interested in the final state after an optimization. Some rare cases may want
|
||||||
Some rare cases may want to use the "before" file for completeness.
|
to use the "before" file for completeness.
|
||||||
|
|
||||||
* `// EMIT_MIR $MIR_PATH.diff` — where `$MIR_PATH` is the filename of the MIR
|
- `// EMIT_MIR $MIR_PATH.diff` — where `$MIR_PATH` is the filename of the MIR
|
||||||
dump, such as `my_test_name.my_function.EarlyOtherwiseBranch`.
|
dump, such as `my_test_name.my_function.EarlyOtherwiseBranch`. Compiletest
|
||||||
Compiletest will diff the `.before.mir` and `.after.mir` files, and compare
|
will diff the `.before.mir` and `.after.mir` files, and compare the diff
|
||||||
the diff output to the expected `.diff` file from the `EMIT_MIR` comment.
|
output to the expected `.diff` file from the `EMIT_MIR` comment.
|
||||||
|
|
||||||
This is useful if you want to see how an optimization changes the MIR.
|
This is useful if you want to see how an optimization changes the MIR.
|
||||||
|
|
||||||
* `// EMIT_MIR $MIR_PATH.dot` — When using specific flags that dump additional
|
- `// EMIT_MIR $MIR_PATH.dot` — When using specific flags that dump additional
|
||||||
MIR data (e.g. `-Z dump-mir-graphviz` to produce `.dot` files), this will
|
MIR data (e.g. `-Z dump-mir-graphviz` to produce `.dot` files), this will
|
||||||
check that the output matches the given file.
|
check that the output matches the given file.
|
||||||
|
|
||||||
|
|
@ -377,35 +383,33 @@ your test, causing separate files to be generated for 32bit and 64bit systems.
|
||||||
|
|
||||||
### `run-make` tests
|
### `run-make` tests
|
||||||
|
|
||||||
> NOTE:
|
> **Note on phasing out `Makefile`s**
|
||||||
|
>
|
||||||
> We are planning to migrate all existing Makefile-based `run-make` tests
|
> We are planning to migrate all existing Makefile-based `run-make` tests
|
||||||
> to Rust recipes. You should not be adding new Makefile-based `run-make`
|
> to Rust programs. You should not be adding new Makefile-based `run-make`
|
||||||
> tests.
|
> tests.
|
||||||
|
>
|
||||||
|
> See <https://github.com/rust-lang/rust/issues/121876>.
|
||||||
|
|
||||||
The tests in [`tests/run-make`] are general-purpose tests using Rust *recipes*,
|
The tests in [`tests/run-make`] are general-purpose tests using Rust *recipes*,
|
||||||
which are small programs allowing arbitrary Rust code such as `rustc`
|
which are small programs (`rmake.rs`) allowing arbitrary Rust code such as
|
||||||
invocations, and is supported by a [`run_make_support`] library. Using Rust
|
`rustc` invocations, and is supported by a [`run_make_support`] library. Using
|
||||||
recipes provide the ultimate in flexibility.
|
Rust recipes provide the ultimate in flexibility.
|
||||||
|
|
||||||
*These should be used as a last resort*. If possible, you should use one of the
|
`run-make` tests should be used if no other test suites better suit your needs.
|
||||||
other test suites.
|
|
||||||
|
|
||||||
If there is some minor feature missing which you need for your test,
|
|
||||||
consider extending compiletest to add a header command for what you need.
|
|
||||||
However, if running a bunch of commands is really what you need,
|
|
||||||
`run-make` is here to the rescue!
|
|
||||||
|
|
||||||
#### Using Rust recipes
|
#### Using Rust recipes
|
||||||
|
|
||||||
Each test should be in a separate directory with a `rmake.rs` Rust program,
|
Each test should be in a separate directory with a `rmake.rs` Rust program,
|
||||||
called the *recipe*. A recipe will be compiled and executed by compiletest
|
called the *recipe*. A recipe will be compiled and executed by compiletest with
|
||||||
with the `run_make_support` library linked in.
|
the `run_make_support` library linked in.
|
||||||
|
|
||||||
If you need new utilities or functionality, consider extending and improving
|
If you need new utilities or functionality, consider extending and improving the
|
||||||
the [`run_make_support`] library.
|
[`run_make_support`] library.
|
||||||
|
|
||||||
Compiletest directives like `//@ only-<target>` or `//@ ignore-<target>` are supported in
|
Compiletest directives like `//@ only-<target>` or `//@ ignore-<target>` are
|
||||||
`rmake.rs`, like in UI tests.
|
supported in `rmake.rs`, like in UI tests. However, revisions or building
|
||||||
|
auxiliary via directives are not currently supported.
|
||||||
|
|
||||||
Two `run-make` tests are ported over to Rust recipes as examples:
|
Two `run-make` tests are ported over to Rust recipes as examples:
|
||||||
|
|
||||||
|
|
@ -426,14 +430,16 @@ Of course, some tests will not successfully *run* in this way.
|
||||||
|
|
||||||
#### Using Makefiles (legacy)
|
#### Using Makefiles (legacy)
|
||||||
|
|
||||||
> NOTE:
|
<div class="warning">
|
||||||
> You should avoid writing new Makefile-based `run-make` tests.
|
You should avoid writing new Makefile-based `run-make` tests.
|
||||||
|
</div>
|
||||||
|
|
||||||
Each test should be in a separate directory with a `Makefile` indicating the
|
Each test should be in a separate directory with a `Makefile` indicating the
|
||||||
commands to run.
|
commands to run.
|
||||||
|
|
||||||
There is a [`tools.mk`] Makefile which you can include which provides a bunch of
|
There is a [`tools.mk`] Makefile which you can include which provides a bunch of
|
||||||
utilities to make it easier to run commands and compare outputs.
|
utilities to make it easier to run commands and compare outputs. Take a look at
|
||||||
Take a look at some of the other tests for some examples on how to get started.
|
some of the other tests for some examples on how to get started.
|
||||||
|
|
||||||
[`tools.mk`]: https://github.com/rust-lang/rust/blob/master/tests/run-make/tools.mk
|
[`tools.mk`]: https://github.com/rust-lang/rust/blob/master/tests/run-make/tools.mk
|
||||||
[`tests/run-make`]: https://github.com/rust-lang/rust/tree/master/tests/run-make
|
[`tests/run-make`]: https://github.com/rust-lang/rust/tree/master/tests/run-make
|
||||||
|
|
@ -442,9 +448,13 @@ Take a look at some of the other tests for some examples on how to get started.
|
||||||
|
|
||||||
### Valgrind tests
|
### Valgrind tests
|
||||||
|
|
||||||
The tests in [`tests/run-pass-valgrind`] are for use with [Valgrind].
|
> **TODO**
|
||||||
These are currently vestigial, as Valgrind is no longer used in CI.
|
>
|
||||||
These may be removed in the future.
|
> Yeet this if we yeet the test suite.
|
||||||
|
|
||||||
|
The tests in [`tests/run-pass-valgrind`] are for use with [Valgrind]. These are
|
||||||
|
currently vestigial, as Valgrind is no longer used in CI. These may be removed
|
||||||
|
in the future.
|
||||||
|
|
||||||
[Valgrind]: https://valgrind.org/
|
[Valgrind]: https://valgrind.org/
|
||||||
[`tests/run-pass-valgrind`]: https://github.com/rust-lang/rust/tree/master/tests/run-pass-valgrind
|
[`tests/run-pass-valgrind`]: https://github.com/rust-lang/rust/tree/master/tests/run-pass-valgrind
|
||||||
|
|
@ -453,9 +463,8 @@ These may be removed in the future.
|
||||||
### Coverage tests
|
### Coverage tests
|
||||||
|
|
||||||
The tests in [`tests/coverage`] are shared by multiple test modes that test
|
The tests in [`tests/coverage`] are shared by multiple test modes that test
|
||||||
coverage instrumentation in different ways.
|
coverage instrumentation in different ways. Running the `coverage` test suite
|
||||||
Running the `coverage` test suite will automatically run each test in all of
|
will automatically run each test in all of the different coverage modes.
|
||||||
the different coverage modes.
|
|
||||||
|
|
||||||
Each mode also has an alias to run the coverage tests in just that mode:
|
Each mode also has an alias to run the coverage tests in just that mode:
|
||||||
|
|
||||||
|
|
@ -471,35 +480,34 @@ Each mode also has an alias to run the coverage tests in just that mode:
|
||||||
./x test coverage-map -- tests/coverage/if.rs # runs the specified test in "coverage-map" mode only
|
./x test coverage-map -- tests/coverage/if.rs # runs the specified test in "coverage-map" mode only
|
||||||
```
|
```
|
||||||
|
|
||||||
---
|
#### `coverage-map` suite
|
||||||
|
|
||||||
In `coverage-map` mode, these tests verify the mappings between source code
|
In `coverage-map` mode, these tests verify the mappings between source code
|
||||||
regions and coverage counters that are emitted by LLVM.
|
regions and coverage counters that are emitted by LLVM. They compile the test
|
||||||
They compile the test with `--emit=llvm-ir`,
|
with `--emit=llvm-ir`, then use a custom tool ([`src/tools/coverage-dump`]) to
|
||||||
then use a custom tool ([`src/tools/coverage-dump`])
|
extract and pretty-print the coverage mappings embedded in the IR. These tests
|
||||||
to extract and pretty-print the coverage mappings embedded in the IR.
|
don't require the profiler runtime, so they run in PR CI jobs and are easy to
|
||||||
These tests don't require the profiler runtime, so they run in PR CI jobs and
|
run/bless locally.
|
||||||
are easy to run/bless locally.
|
|
||||||
|
|
||||||
These coverage map tests can be sensitive to changes in MIR lowering or MIR
|
These coverage map tests can be sensitive to changes in MIR lowering or MIR
|
||||||
optimizations, producing mappings that are different but produce identical
|
optimizations, producing mappings that are different but produce identical
|
||||||
coverage reports.
|
coverage reports.
|
||||||
|
|
||||||
As a rule of thumb, any PR that doesn't change coverage-specific
|
As a rule of thumb, any PR that doesn't change coverage-specific code should
|
||||||
code should **feel free to re-bless** the `coverage-map` tests as necessary,
|
**feel free to re-bless** the `coverage-map` tests as necessary, without
|
||||||
without worrying about the actual changes, as long as the `coverage-run` tests
|
worrying about the actual changes, as long as the `coverage-run` tests still
|
||||||
still pass.
|
pass.
|
||||||
|
|
||||||
---
|
#### `coverage-run` suite
|
||||||
|
|
||||||
In `coverage-run` mode, these tests perform an end-to-end test of coverage reporting.
|
In `coverage-run` mode, these tests perform an end-to-end test of coverage
|
||||||
They compile a test program with coverage instrumentation, run that program to
|
reporting. They compile a test program with coverage instrumentation, run that
|
||||||
produce raw coverage data, and then use LLVM tools to process that data into a
|
program to produce raw coverage data, and then use LLVM tools to process that
|
||||||
human-readable code coverage report.
|
data into a human-readable code coverage report.
|
||||||
|
|
||||||
Instrumented binaries need to be linked against the LLVM profiler runtime,
|
Instrumented binaries need to be linked against the LLVM profiler runtime, so
|
||||||
so `coverage-run` tests are **automatically skipped**
|
`coverage-run` tests are **automatically skipped** unless the profiler runtime
|
||||||
unless the profiler runtime is enabled in `config.toml`:
|
is enabled in `config.toml`:
|
||||||
|
|
||||||
```toml
|
```toml
|
||||||
# config.toml
|
# config.toml
|
||||||
|
|
@ -507,10 +515,10 @@ unless the profiler runtime is enabled in `config.toml`:
|
||||||
profiler = true
|
profiler = true
|
||||||
```
|
```
|
||||||
|
|
||||||
This also means that they typically don't run in PR CI jobs,
|
This also means that they typically don't run in PR CI jobs, though they do run
|
||||||
though they do run as part of the full set of CI jobs used for merging.
|
as part of the full set of CI jobs used for merging.
|
||||||
|
|
||||||
---
|
#### `coverage-run-rustdoc` suite
|
||||||
|
|
||||||
The tests in [`tests/coverage-run-rustdoc`] also run instrumented doctests and
|
The tests in [`tests/coverage-run-rustdoc`] also run instrumented doctests and
|
||||||
include them in the coverage report. This avoids having to build rustdoc when
|
include them in the coverage report. This avoids having to build rustdoc when
|
||||||
|
|
@ -522,27 +530,30 @@ only running the main `coverage` suite.
|
||||||
|
|
||||||
### Crashes tests
|
### Crashes tests
|
||||||
|
|
||||||
[`tests/crashes`] serve as a collection of tests that are expected to cause the compiler to ICE, panic
|
[`tests/crashes`] serve as a collection of tests that are expected to cause the
|
||||||
or crash in some other way, so that accidental fixes are tracked. This was formally done at
|
compiler to ICE, panic or crash in some other way, so that accidental fixes are
|
||||||
<https://github.com/rust-lang/glacier> but doing it inside the rust-lang/rust testsuite is more
|
tracked. This was formally done at <https://github.com/rust-lang/glacier> but
|
||||||
convenient.
|
doing it inside the rust-lang/rust testsuite is more convenient.
|
||||||
|
|
||||||
It is imperative that a test in the suite causes rustc to ICE, panic or crash crash in some other
|
It is imperative that a test in the suite causes rustc to ICE, panic or crash
|
||||||
way. A test will "pass" if rustc exits with an exit status other than 1 or 0.
|
crash in some other way. A test will "pass" if rustc exits with an exit status
|
||||||
|
other than 1 or 0.
|
||||||
|
|
||||||
If you want to see verbose stdout/stderr, you need to set `COMPILETEST_VERBOSE_CRASHES=1`, e.g.
|
If you want to see verbose stdout/stderr, you need to set
|
||||||
|
`COMPILETEST_VERBOSE_CRASHES=1`, e.g.
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
$ COMPILETEST_VERBOSE_CRASHES=1 ./x test tests/crashes/999999.rs --stage 1
|
$ COMPILETEST_VERBOSE_CRASHES=1 ./x test tests/crashes/999999.rs --stage 1
|
||||||
```
|
```
|
||||||
|
|
||||||
When adding crashes from <https://github.com/rust-lang/rust/issues>, the issue number should be
|
When adding crashes from <https://github.com/rust-lang/rust/issues>, the issue
|
||||||
noted in the file name (`12345.rs` should suffice) and also inside the file include a `//@ known-bug
|
number should be noted in the file name (`12345.rs` should suffice) and also
|
||||||
#4321` directive.
|
inside the file include a `//@ known-bug: #4321` directive.
|
||||||
|
|
||||||
If you happen to fix one of the crashes, please move it to a fitting subdirectory in `tests/ui` and
|
If you happen to fix one of the crashes, please move it to a fitting
|
||||||
give it a meaningful name. Please add a doc comment at the top of the file explaining why this test
|
subdirectory in `tests/ui` and give it a meaningful name. Please add a doc
|
||||||
exists, even better if you can briefly explain how the example causes rustc to crash previously and
|
comment at the top of the file explaining why this test exists, even better if
|
||||||
|
you can briefly explain how the example causes rustc to crash previously and
|
||||||
what was done to prevent rustc to ICE/panic/crash.
|
what was done to prevent rustc to ICE/panic/crash.
|
||||||
|
|
||||||
Adding
|
Adding
|
||||||
|
|
@ -554,24 +565,25 @@ Fixes #MMMMM
|
||||||
|
|
||||||
to the description of your pull request will ensure the corresponding tickets be closed
|
to the description of your pull request will ensure the corresponding tickets be closed
|
||||||
automatically upon merge.
|
automatically upon merge.
|
||||||
Make sure that your fix actually fixes the root cause of the issue and not just a subset first.
|
|
||||||
The issue numbers can be found in the file name or the `//@ known-bug`
|
Make sure that your fix actually fixes the root cause of the issue and not just
|
||||||
directive inside the test file.
|
a subset first. The issue numbers can be found in the file name or the `//@
|
||||||
|
known-bug` directive inside the test file.
|
||||||
|
|
||||||
[`tests/crashes`]: https://github.com/rust-lang/rust/tree/master/tests/crashes
|
[`tests/crashes`]: https://github.com/rust-lang/rust/tree/master/tests/crashes
|
||||||
|
|
||||||
## Building auxiliary crates
|
## Building auxiliary crates
|
||||||
|
|
||||||
It is common that some tests require additional auxiliary crates to be compiled.
|
It is common that some tests require additional auxiliary crates to be compiled.
|
||||||
There are multiple [headers](headers.md) to assist with that:
|
There are multiple [directives](directives.md) to assist with that:
|
||||||
|
|
||||||
* `aux-build`
|
- `aux-build`
|
||||||
* `aux-crate`
|
- `aux-crate`
|
||||||
* `aux-bin`
|
- `aux-bin`
|
||||||
* `aux-codegen-backend`
|
- `aux-codegen-backend`
|
||||||
|
|
||||||
`aux-build` will build a separate crate from the named source file.
|
`aux-build` will build a separate crate from the named source file. The source
|
||||||
The source file should be in a directory called `auxiliary` beside the test file.
|
file should be in a directory called `auxiliary` beside the test file.
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
//@ aux-build: my-helper.rs
|
//@ aux-build: my-helper.rs
|
||||||
|
|
@ -581,16 +593,14 @@ extern crate my_helper;
|
||||||
```
|
```
|
||||||
|
|
||||||
The aux crate will be built as a dylib if possible (unless on a platform that
|
The aux crate will be built as a dylib if possible (unless on a platform that
|
||||||
does not support them, or the `no-prefer-dynamic` header is specified in the
|
does not support them, or the `no-prefer-dynamic` header is specified in the aux
|
||||||
aux file).
|
file). The `-L` flag is used to find the extern crates.
|
||||||
The `-L` flag is used to find the extern crates.
|
|
||||||
|
|
||||||
`aux-crate` is very similar to `aux-build`; however, it uses the `--extern`
|
`aux-crate` is very similar to `aux-build`. However, it uses the `--extern` flag
|
||||||
flag to link to the extern crate.
|
to link to the extern crate to make the crate be available as an extern prelude.
|
||||||
That allows you to specify the additional syntax of the `--extern` flag, such
|
That allows you to specify the additional syntax of the `--extern` flag, such as
|
||||||
as renaming a dependency.
|
renaming a dependency. For example, `// aux-crate:foo=bar.rs` will compile
|
||||||
For example, `// aux-crate:foo=bar.rs` will compile `auxiliary/bar.rs` and
|
`auxiliary/bar.rs` and make it available under then name `foo` within the test.
|
||||||
make it available under then name `foo` within the test.
|
|
||||||
This is similar to how Cargo does dependency renaming.
|
This is similar to how Cargo does dependency renaming.
|
||||||
|
|
||||||
`aux-bin` is similar to `aux-build` but will build a binary instead of a
|
`aux-bin` is similar to `aux-build` but will build a binary instead of a
|
||||||
|
|
@ -605,8 +615,9 @@ for tests in `tests/ui-fulldeps`, since it requires the use of compiler crates.
|
||||||
|
|
||||||
If you want a proc-macro dependency, then there currently is some ceremony
|
If you want a proc-macro dependency, then there currently is some ceremony
|
||||||
needed.
|
needed.
|
||||||
Place the proc-macro itself in a file like `auxiliary/my-proc-macro.rs`
|
|
||||||
with the following structure:
|
Place the proc-macro itself in a file like `auxiliary/my-proc-macro.rs` with the
|
||||||
|
following structure:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
//@ force-host
|
//@ force-host
|
||||||
|
|
@ -623,10 +634,10 @@ pub fn foo(input: TokenStream) -> TokenStream {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
The `force-host` is needed because proc-macros are loaded in the host
|
The `force-host` is needed because proc-macros are loaded in the host compiler,
|
||||||
compiler, and `no-prefer-dynamic` is needed to tell compiletest to not use
|
and `no-prefer-dynamic` is needed to tell compiletest to not use
|
||||||
`prefer-dynamic` which is not compatible with proc-macros.
|
`prefer-dynamic` which is not compatible with proc-macros. The `#![crate_type]`
|
||||||
The `#![crate_type]` attribute is needed to specify the correct crate-type.
|
attribute is needed to specify the correct crate-type.
|
||||||
|
|
||||||
Then in your test, you can build with `aux-build`:
|
Then in your test, you can build with `aux-build`:
|
||||||
|
|
||||||
|
|
@ -643,21 +654,20 @@ fn main() {
|
||||||
|
|
||||||
## Revisions
|
## Revisions
|
||||||
|
|
||||||
Revisions allow a single test file to be used for multiple tests.
|
Revisions allow a single test file to be used for multiple tests. This is done
|
||||||
This is done by adding a special header at the top of the file:
|
by adding a special directive at the top of the file:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
//@ revisions: foo bar baz
|
//@ revisions: foo bar baz
|
||||||
```
|
```
|
||||||
|
|
||||||
This will result in the test being compiled (and tested) three times,
|
This will result in the test being compiled (and tested) three times, once with
|
||||||
once with `--cfg foo`, once with `--cfg bar`, and once with `--cfg
|
`--cfg foo`, once with `--cfg bar`, and once with `--cfg baz`. You can therefore
|
||||||
baz`.
|
use `#[cfg(foo)]` etc within the test to tweak each of these results.
|
||||||
You can therefore use `#[cfg(foo)]` etc within the test to tweak
|
|
||||||
each of these results.
|
|
||||||
|
|
||||||
You can also customize headers and expected error messages to a particular
|
You can also customize directives and expected error messages to a particular
|
||||||
revision. To do this, add `[revision-name]` after the `//` comment, like so:
|
revision. To do this, add `[revision-name]` after the `//@` for directives, and
|
||||||
|
after `//` for UI error annotations, like so:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
// A flag to pass in only for cfg `foo`:
|
// A flag to pass in only for cfg `foo`:
|
||||||
|
|
@ -677,8 +687,8 @@ also registered as an additional prefix for FileCheck directives:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
//@ revisions: NORMAL COVERAGE
|
//@ revisions: NORMAL COVERAGE
|
||||||
//@ [COVERAGE] compile-flags: -Cinstrument-coverage
|
//@[COVERAGE] compile-flags: -Cinstrument-coverage
|
||||||
//@ [COVERAGE] needs-profiler-support
|
//@[COVERAGE] needs-profiler-support
|
||||||
|
|
||||||
// COVERAGE: @__llvm_coverage_mapping
|
// COVERAGE: @__llvm_coverage_mapping
|
||||||
// NORMAL-NOT: @__llvm_coverage_mapping
|
// NORMAL-NOT: @__llvm_coverage_mapping
|
||||||
|
|
@ -687,45 +697,46 @@ also registered as an additional prefix for FileCheck directives:
|
||||||
fn main() {}
|
fn main() {}
|
||||||
```
|
```
|
||||||
|
|
||||||
Note that not all headers have meaning when customized to a revision.
|
Note that not all directives have meaning when customized to a revision. For
|
||||||
For example, the `ignore-test` header (and all "ignore" headers)
|
example, the `ignore-test` directives (and all "ignore" directives) currently
|
||||||
currently only apply to the test as a whole, not to particular
|
only apply to the test as a whole, not to particular revisions. The only
|
||||||
revisions. The only headers that are intended to really work when
|
directives that are intended to really work when customized to a revision are
|
||||||
customized to a revision are error patterns and compiler flags.
|
error patterns and compiler flags.
|
||||||
|
|
||||||
<!-- date-check jul 2023 -->
|
<!-- date-check jul 2023 -->
|
||||||
Following is classes of tests that support revisions:
|
The following test suites support revisions:
|
||||||
- UI
|
|
||||||
|
- ui
|
||||||
- assembly
|
- assembly
|
||||||
- codegen
|
- codegen
|
||||||
- coverage
|
- coverage
|
||||||
- debuginfo
|
- debuginfo
|
||||||
- rustdoc UI tests
|
- rustdoc UI tests
|
||||||
- incremental (these are special in that they inherently cannot be run in parallel)
|
- incremental (these are special in that they inherently cannot be run in
|
||||||
|
parallel)
|
||||||
|
|
||||||
### Ignoring unused revision names
|
### Ignoring unused revision names
|
||||||
|
|
||||||
Normally, revision names mentioned in other headers and error annotations must
|
Normally, revision names mentioned in other directives and error annotations
|
||||||
correspond to an actual revision declared in a `revisions` header. This is
|
must correspond to an actual revision declared in a `revisions` directive. This is
|
||||||
enforced by an `./x test tidy` check.
|
enforced by an `./x test tidy` check.
|
||||||
|
|
||||||
If a revision name needs to be temporarily removed from the revision list for
|
If a revision name needs to be temporarily removed from the revision list for
|
||||||
some reason, the above check can be suppressed by adding the revision name to
|
some reason, the above check can be suppressed by adding the revision name to an
|
||||||
an `//@ unused-revision-names:` header instead.
|
`//@ unused-revision-names:` header instead.
|
||||||
|
|
||||||
Specifying an unused name of `*` (i.e. `//@ unused-revision-names: *`) will
|
Specifying an unused name of `*` (i.e. `//@ unused-revision-names: *`) will
|
||||||
permit any unused revision name to be mentioned.
|
permit any unused revision name to be mentioned.
|
||||||
|
|
||||||
## Compare modes
|
## Compare modes
|
||||||
|
|
||||||
Compiletest can be run in different modes, called _compare modes_, which can
|
Compiletest can be run in different modes, called _compare modes_, which can be
|
||||||
be used to compare the behavior of all tests with different compiler flags
|
used to compare the behavior of all tests with different compiler flags enabled.
|
||||||
enabled.
|
|
||||||
This can help highlight what differences might appear with certain flags, and
|
This can help highlight what differences might appear with certain flags, and
|
||||||
check for any problems that might arise.
|
check for any problems that might arise.
|
||||||
|
|
||||||
To run the tests in a different mode, you need to pass the `--compare-mode`
|
To run the tests in a different mode, you need to pass the `--compare-mode` CLI
|
||||||
CLI flag:
|
flag:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test tests/ui --compare-mode=chalk
|
./x test tests/ui --compare-mode=chalk
|
||||||
|
|
@ -733,10 +744,12 @@ CLI flag:
|
||||||
|
|
||||||
The possible compare modes are:
|
The possible compare modes are:
|
||||||
|
|
||||||
* `polonius` — Runs with Polonius with `-Zpolonius`.
|
- `polonius` — Runs with Polonius with `-Zpolonius`.
|
||||||
* `chalk` — Runs with Chalk with `-Zchalk`.
|
- `chalk` — Runs with Chalk with `-Zchalk`.
|
||||||
* `split-dwarf` — Runs with unpacked split-DWARF with `-Csplit-debuginfo=unpacked`.
|
- `split-dwarf` — Runs with unpacked split-DWARF with
|
||||||
* `split-dwarf-single` — Runs with packed split-DWARF with `-Csplit-debuginfo=packed`.
|
`-Csplit-debuginfo=unpacked`.
|
||||||
|
- `split-dwarf-single` — Runs with packed split-DWARF with
|
||||||
|
`-Csplit-debuginfo=packed`.
|
||||||
|
|
||||||
See [UI compare modes](ui.md#compare-modes) for more information about how UI
|
See [UI compare modes](ui.md#compare-modes) for more information about how UI
|
||||||
tests support different output for different modes.
|
tests support different output for different modes.
|
||||||
|
|
@ -744,10 +757,9 @@ tests support different output for different modes.
|
||||||
In CI, compare modes are only used in one Linux builder, and only with the
|
In CI, compare modes are only used in one Linux builder, and only with the
|
||||||
following settings:
|
following settings:
|
||||||
|
|
||||||
* `tests/debuginfo`: Uses `split-dwarf` mode.
|
- `tests/debuginfo`: Uses `split-dwarf` mode. This helps ensure that none of the
|
||||||
This helps ensure that none of the debuginfo tests are affected when
|
debuginfo tests are affected when enabling split-DWARF.
|
||||||
enabling split-DWARF.
|
|
||||||
|
|
||||||
Note that compare modes are separate to [revisions](#revisions).
|
Note that compare modes are separate to [revisions](#revisions). All revisions
|
||||||
All revisions are tested when running `./x test tests/ui`, however
|
are tested when running `./x test tests/ui`, however compare-modes must be
|
||||||
compare-modes must be manually run individually via the `--compare-mode` flag.
|
manually run individually via the `--compare-mode` flag.
|
||||||
|
|
|
||||||
|
|
@ -1,10 +1,10 @@
|
||||||
# Crater
|
# Crater
|
||||||
|
|
||||||
[Crater](https://github.com/rust-lang/crater) is a tool for compiling
|
[Crater](https://github.com/rust-lang/crater) is a tool for compiling and
|
||||||
and running tests for _every_ crate on [crates.io](https://crates.io) (and a
|
running tests for _every_ crate on [crates.io](https://crates.io) (and a few on
|
||||||
few on GitHub). It is mainly used for checking the extent of breakage when
|
GitHub). It is mainly used for checking the extent of breakage when implementing
|
||||||
implementing potentially breaking changes and ensuring lack of breakage by
|
potentially breaking changes and ensuring lack of breakage by running beta vs
|
||||||
running beta vs stable compiler versions.
|
stable compiler versions.
|
||||||
|
|
||||||
## When to run Crater
|
## When to run Crater
|
||||||
|
|
||||||
|
|
@ -15,16 +15,16 @@ or could cause breakage. If you are unsure, feel free to ask your PR's reviewer.
|
||||||
|
|
||||||
The rust team maintains a few machines that can be used for running crater runs
|
The rust team maintains a few machines that can be used for running crater runs
|
||||||
on the changes introduced by a PR. If your PR needs a crater run, leave a
|
on the changes introduced by a PR. If your PR needs a crater run, leave a
|
||||||
comment for the triage team in the PR thread. Please inform the team whether
|
comment for the triage team in the PR thread. Please inform the team whether you
|
||||||
you require a "check-only" crater run, a "build only" crater run, or a
|
require a "check-only" crater run, a "build only" crater run, or a
|
||||||
"build-and-test" crater run. The difference is primarily in time; the
|
"build-and-test" crater run. The difference is primarily in time; the
|
||||||
conservative (if you're not sure) option is to go for the build-and-test run.
|
conservative (if you're not sure) option is to go for the build-and-test run. If
|
||||||
If making changes that will only have an effect at compile-time (e.g.,
|
making changes that will only have an effect at compile-time (e.g., implementing
|
||||||
implementing a new trait) then you only need a check run.
|
a new trait) then you only need a check run.
|
||||||
|
|
||||||
Your PR will be enqueued by the triage team and the results will be posted when
|
Your PR will be enqueued by the triage team and the results will be posted when
|
||||||
they are ready. Check runs will take around ~3-4 days, with the other two
|
they are ready. Check runs will take around ~3-4 days, with the other two taking
|
||||||
taking 5-6 days on average.
|
5-6 days on average.
|
||||||
|
|
||||||
While crater is really useful, it is also important to be aware of a few
|
While crater is really useful, it is also important to be aware of a few
|
||||||
caveats:
|
caveats:
|
||||||
|
|
@ -37,9 +37,9 @@ caveats:
|
||||||
- Crater only runs Linux builds on x86_64. Thus, other architectures and
|
- Crater only runs Linux builds on x86_64. Thus, other architectures and
|
||||||
platforms are not tested. Critically, this includes Windows.
|
platforms are not tested. Critically, this includes Windows.
|
||||||
|
|
||||||
- Many crates are not tested. This could be for a lot of reasons, including
|
- Many crates are not tested. This could be for a lot of reasons, including that
|
||||||
that the crate doesn't compile any more (e.g. used old nightly features),
|
the crate doesn't compile any more (e.g. used old nightly features), has
|
||||||
has broken or flaky tests, requires network access, or other reasons.
|
broken or flaky tests, requires network access, or other reasons.
|
||||||
|
|
||||||
- Before crater can be run, `@bors try` needs to succeed in building artifacts.
|
- Before crater can be run, `@bors try` needs to succeed in building artifacts.
|
||||||
This means that if your code doesn't compile, you cannot run crater.
|
This means that if your code doesn't compile, you cannot run crater.
|
||||||
|
|
|
||||||
|
|
@ -2,6 +2,8 @@
|
||||||
|
|
||||||
<!-- toc -->
|
<!-- toc -->
|
||||||
|
|
||||||
|
> **FIXME(jieyouxu)** completely revise this chapter.
|
||||||
|
|
||||||
Header commands are special comments that tell compiletest how to build and
|
Header commands are special comments that tell compiletest how to build and
|
||||||
interpret a test.
|
interpret a test.
|
||||||
They must appear before the Rust source in the test.
|
They must appear before the Rust source in the test.
|
||||||
|
|
@ -0,0 +1,28 @@
|
||||||
|
# Ecosystem testing
|
||||||
|
|
||||||
|
Rust tests integration with real-world code in the ecosystem to catch
|
||||||
|
regressions and make informed decisions about the evolution of the language.
|
||||||
|
|
||||||
|
## Testing methods
|
||||||
|
|
||||||
|
### Crater
|
||||||
|
|
||||||
|
Crater is a tool which runs tests on many thousands of public projects. This
|
||||||
|
tool has its own separate infrastructure for running, and is not run as part of
|
||||||
|
CI. See the [Crater chapter](crater.md) for more details.
|
||||||
|
|
||||||
|
### `cargotest`
|
||||||
|
|
||||||
|
`cargotest` is a small tool which runs `cargo test` on a few sample projects
|
||||||
|
(such as `servo`, `ripgrep`, `tokei`, etc.). This runs as part of CI and ensures
|
||||||
|
there aren't any significant regressions.
|
||||||
|
|
||||||
|
> Example: `./x test src/tools/cargotest`
|
||||||
|
|
||||||
|
### Large OSS Project builders
|
||||||
|
|
||||||
|
We have CI jobs that build large open-source Rust projects that are used as
|
||||||
|
regression tests in CI. Our integration jobs build the following projects:
|
||||||
|
|
||||||
|
- [Fuchsia](fuchsia.md)
|
||||||
|
- [Rust for Linux](rust-for-linux.md)
|
||||||
|
|
@ -36,7 +36,7 @@ See the [Testing with Docker](docker.md) chapter for more details on how to run
|
||||||
and debug jobs with Docker.
|
and debug jobs with Docker.
|
||||||
|
|
||||||
Note that a Fuchsia checkout is *large* – as of this writing, a checkout and
|
Note that a Fuchsia checkout is *large* – as of this writing, a checkout and
|
||||||
build takes 46G of space – and as you might imagine, it takes awhile to
|
build takes 46G of space – and as you might imagine, it takes a while to
|
||||||
complete.
|
complete.
|
||||||
|
|
||||||
### Modifying the Fuchsia checkout
|
### Modifying the Fuchsia checkout
|
||||||
|
|
@ -65,11 +65,12 @@ to add this to your `$PATH` for some workflows.
|
||||||
|
|
||||||
There are a few `fx` subcommands that are relevant, including:
|
There are a few `fx` subcommands that are relevant, including:
|
||||||
|
|
||||||
* `fx set` accepts build arguments, writes them to `out/default/args.gn`, and runs GN.
|
- `fx set` accepts build arguments, writes them to `out/default/args.gn`, and
|
||||||
* `fx build` builds the Fuchsia project using Ninja. It will automatically pick
|
runs GN.
|
||||||
|
- `fx build` builds the Fuchsia project using Ninja. It will automatically pick
|
||||||
up changes to build arguments and rerun GN. By default it builds everything,
|
up changes to build arguments and rerun GN. By default it builds everything,
|
||||||
but it also accepts target paths to build specific targets (see below).
|
but it also accepts target paths to build specific targets (see below).
|
||||||
* `fx clippy` runs Clippy on specific Rust targets (or all of them). We use this
|
- `fx clippy` runs Clippy on specific Rust targets (or all of them). We use this
|
||||||
in the Rust CI build to avoid running codegen on most Rust targets. Underneath
|
in the Rust CI build to avoid running codegen on most Rust targets. Underneath
|
||||||
it invokes Ninja, just like `fx build`. The clippy results are saved in json
|
it invokes Ninja, just like `fx build`. The clippy results are saved in json
|
||||||
files inside the build output directory before being printed.
|
files inside the build output directory before being printed.
|
||||||
|
|
@ -94,8 +95,8 @@ and can also be used in `fx build`.
|
||||||
|
|
||||||
#### Modifying compiler flags
|
#### Modifying compiler flags
|
||||||
|
|
||||||
You can put custom compiler flags inside a GN `config` that is added to a target.
|
You can put custom compiler flags inside a GN `config` that is added to a
|
||||||
As a simple example:
|
target. As a simple example:
|
||||||
|
|
||||||
```
|
```
|
||||||
config("everybody_loops") {
|
config("everybody_loops") {
|
||||||
|
|
@ -162,6 +163,6 @@ rustc book][platform-support].
|
||||||
[`//build/config:compiler`]: https://cs.opensource.google/fuchsia/fuchsia/+/main:build/config/BUILD.gn;l=121;drc=c26c473bef93b33117ae417893118907a026fec7
|
[`//build/config:compiler`]: https://cs.opensource.google/fuchsia/fuchsia/+/main:build/config/BUILD.gn;l=121;drc=c26c473bef93b33117ae417893118907a026fec7
|
||||||
[build system]: https://fuchsia.dev/fuchsia-src/development/build/build_system
|
[build system]: https://fuchsia.dev/fuchsia-src/development/build/build_system
|
||||||
|
|
||||||
[^loc]: As of June 2024, Fuchsia had about 2 million lines of first-party Rust code
|
[^loc]: As of June 2024, Fuchsia had about 2 million lines of first-party Rust
|
||||||
and a roughly equal amount of third-party code, as counted by tokei (excluding
|
code and a roughly equal amount of third-party code, as counted by tokei
|
||||||
comments and blanks).
|
(excluding comments and blanks).
|
||||||
|
|
|
||||||
|
|
@ -1,49 +0,0 @@
|
||||||
# Integration testing
|
|
||||||
|
|
||||||
Rust tests integration with real-world code to catch regressions and make
|
|
||||||
informed decisions about the evolution of the language.
|
|
||||||
|
|
||||||
## Testing methods
|
|
||||||
|
|
||||||
### Crater
|
|
||||||
|
|
||||||
Crater is a tool which runs tests on many thousands of public projects. This
|
|
||||||
tool has its own separate infrastructure for running, and is not run as part of
|
|
||||||
CI. See the [Crater chapter](crater.md) for more details.
|
|
||||||
|
|
||||||
### Cargo test
|
|
||||||
|
|
||||||
`cargotest` is a small tool which runs `cargo test` on a few sample projects
|
|
||||||
(such as `servo`, `ripgrep`, `tokei`, etc.).
|
|
||||||
This runs as part of CI and ensures there aren't any significant regressions.
|
|
||||||
|
|
||||||
> Example: `./x test src/tools/cargotest`
|
|
||||||
|
|
||||||
### Integration builders
|
|
||||||
|
|
||||||
Integration jobs build large open-source Rust projects that are used as
|
|
||||||
regression tests in CI. Our integration jobs build the following projects:
|
|
||||||
|
|
||||||
- [Fuchsia](fuchsia.md)
|
|
||||||
- [Rust for Linux](rust-for-linux.md)
|
|
||||||
|
|
||||||
## A note about terminology
|
|
||||||
|
|
||||||
The term "integration testing" can be used to mean many things. Many of the
|
|
||||||
compiletest tests within the Rust repo could be justifiably called integration
|
|
||||||
tests, because they test the integration of many parts of the compiler, or test
|
|
||||||
the integration of the compiler with other external tools. Calling all of them
|
|
||||||
integration tests would not be very helpful, especially since those kinds of
|
|
||||||
tests already have their own specialized names.
|
|
||||||
|
|
||||||
We use the term "integration" here to mean integrating the Rust compiler and
|
|
||||||
toolchain with the ecosystem of Rust projects that depend on it. This is partly
|
|
||||||
for lack of a better term, but it also reflects a difference in testing approach
|
|
||||||
from other projects and the comparative advantage it implies.
|
|
||||||
|
|
||||||
The Rust compiler is part of the ecosystem, and the ecosystem is in many cases
|
|
||||||
part of Rust, both in terms of libraries it uses and in terms of the efforts of many
|
|
||||||
contributors who come to "scratch their own itch". Finally, because Rust has the
|
|
||||||
ability to do integration testing at such a broad scale, it shortens development
|
|
||||||
cycles by finding defects earlier.
|
|
||||||
|
|
||||||
|
|
@ -2,10 +2,10 @@
|
||||||
|
|
||||||
<!-- toc -->
|
<!-- toc -->
|
||||||
|
|
||||||
The Rust project runs a wide variety of different tests, orchestrated by
|
The Rust project runs a wide variety of different tests, orchestrated by the
|
||||||
the build system (`./x test`).
|
build system (`./x test`). This section gives a brief overview of the different
|
||||||
This section gives a brief overview of the different testing tools.
|
testing tools. Subsequent chapters dive into [running tests](running.md) and
|
||||||
Subsequent chapters dive into [running tests](running.md) and [adding new tests](adding.md).
|
[adding new tests](adding.md).
|
||||||
|
|
||||||
## Kinds of tests
|
## Kinds of tests
|
||||||
|
|
||||||
|
|
@ -14,9 +14,13 @@ Almost all of them are driven by `./x test`, with some exceptions noted below.
|
||||||
|
|
||||||
### Compiletest
|
### Compiletest
|
||||||
|
|
||||||
The main test harness for testing the compiler itself is a tool called [compiletest].
|
The main test harness for testing the compiler itself is a tool called
|
||||||
It supports running different styles of tests, called *test suites*.
|
[compiletest].
|
||||||
The tests are all located in the [`tests`] directory.
|
|
||||||
|
[compiletest] supports running different styles of tests, organized into *test
|
||||||
|
suites*. A *test mode* may provide common presets/behavior for a set of *test
|
||||||
|
suites*. [compiletest]-supported tests are located in the [`tests`] directory.
|
||||||
|
|
||||||
The [Compiletest chapter][compiletest] goes into detail on how to use this tool.
|
The [Compiletest chapter][compiletest] goes into detail on how to use this tool.
|
||||||
|
|
||||||
> Example: `./x test tests/ui`
|
> Example: `./x test tests/ui`
|
||||||
|
|
@ -26,10 +30,10 @@ The [Compiletest chapter][compiletest] goes into detail on how to use this tool.
|
||||||
|
|
||||||
### Package tests
|
### Package tests
|
||||||
|
|
||||||
The standard library and many of the compiler packages include typical Rust `#[test]`
|
The standard library and many of the compiler packages include typical Rust
|
||||||
unit tests, integration tests, and documentation tests.
|
`#[test]` unit tests, integration tests, and documentation tests. You can pass a
|
||||||
You can pass a path to `x` to almost any package in the `library` or `compiler` directory,
|
path to `./x test` for almost any package in the `library/` or `compiler/`
|
||||||
and `x` will essentially run `cargo test` on that package.
|
directory, and `x` will essentially run `cargo test` on that package.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
|
|
||||||
|
|
@ -39,25 +43,25 @@ Examples:
|
||||||
| `./x test library/core` | Runs tests on `core` only |
|
| `./x test library/core` | Runs tests on `core` only |
|
||||||
| `./x test compiler/rustc_data_structures` | Runs tests on `rustc_data_structures` |
|
| `./x test compiler/rustc_data_structures` | Runs tests on `rustc_data_structures` |
|
||||||
|
|
||||||
The standard library relies very heavily on documentation tests to cover its functionality.
|
The standard library relies very heavily on documentation tests to cover its
|
||||||
However, unit tests and integration tests can also be used as needed.
|
functionality. However, unit tests and integration tests can also be used as
|
||||||
Almost all of the compiler packages have doctests disabled.
|
needed. Almost all of the compiler packages have doctests disabled.
|
||||||
|
|
||||||
All standard library and compiler unit tests are placed in separate `tests` file
|
All standard library and compiler unit tests are placed in separate `tests` file
|
||||||
(which is enforced in [tidy][tidy-unit-tests]).
|
(which is enforced in [tidy][tidy-unit-tests]). This ensures that when the test
|
||||||
This ensures that when the test file is changed, the crate does not need to be recompiled.
|
file is changed, the crate does not need to be recompiled. For example:
|
||||||
For example:
|
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
#[cfg(test)]
|
#[cfg(test)]
|
||||||
mod tests;
|
mod tests;
|
||||||
```
|
```
|
||||||
|
|
||||||
If it wasn't done this way,
|
If it wasn't done this way, and you were working on something like `core`, that
|
||||||
and you were working on something like `core`,
|
would require recompiling the entire standard library, and the entirety of
|
||||||
that would require recompiling the entire standard library, and the entirety of `rustc`.
|
`rustc`.
|
||||||
|
|
||||||
`./x test` includes some CLI options for controlling the behavior with these tests:
|
`./x test` includes some CLI options for controlling the behavior with these
|
||||||
|
package tests:
|
||||||
|
|
||||||
* `--doc` — Only runs documentation tests in the package.
|
* `--doc` — Only runs documentation tests in the package.
|
||||||
* `--no-doc` — Run all tests *except* documentation tests.
|
* `--no-doc` — Run all tests *except* documentation tests.
|
||||||
|
|
@ -66,16 +70,18 @@ that would require recompiling the entire standard library, and the entirety of
|
||||||
|
|
||||||
### Tidy
|
### Tidy
|
||||||
|
|
||||||
Tidy is a custom tool used for validating source code style and formatting conventions,
|
Tidy is a custom tool used for validating source code style and formatting
|
||||||
such as rejecting long lines.
|
conventions, such as rejecting long lines. There is more information in the
|
||||||
There is more information in the [section on coding conventions](../conventions.md#formatting).
|
[section on coding conventions](../conventions.md#formatting).
|
||||||
|
|
||||||
|
> Examples: `./x test tidy`
|
||||||
|
|
||||||
> Example: `./x test tidy`
|
|
||||||
|
|
||||||
### Formatting
|
### Formatting
|
||||||
|
|
||||||
Rustfmt is integrated with the build system to enforce uniform style across the compiler.
|
Rustfmt is integrated with the build system to enforce uniform style across the
|
||||||
The formatting check is automatically run by the Tidy tool mentioned above.
|
compiler. The formatting check is automatically run by the Tidy tool mentioned
|
||||||
|
above.
|
||||||
|
|
||||||
Examples:
|
Examples:
|
||||||
|
|
||||||
|
|
@ -87,10 +93,10 @@ Examples:
|
||||||
|
|
||||||
### Book documentation tests
|
### Book documentation tests
|
||||||
|
|
||||||
All of the books that are published have their own tests,
|
All of the books that are published have their own tests, primarily for
|
||||||
primarily for validating that the Rust code examples pass.
|
validating that the Rust code examples pass. Under the hood, these are
|
||||||
Under the hood, these are essentially using `rustdoc --test` on the markdown files.
|
essentially using `rustdoc --test` on the markdown files. The tests can be run
|
||||||
The tests can be run by passing a path to a book to `./x test`.
|
by passing a path to a book to `./x test`.
|
||||||
|
|
||||||
> Example: `./x test src/doc/book`
|
> Example: `./x test src/doc/book`
|
||||||
|
|
||||||
|
|
@ -106,47 +112,48 @@ This requires building all of the documentation, which might take a while.
|
||||||
|
|
||||||
### Dist check
|
### Dist check
|
||||||
|
|
||||||
`distcheck` verifies that the source distribution tarball created by the build system
|
`distcheck` verifies that the source distribution tarball created by the build
|
||||||
will unpack, build, and run all tests.
|
system will unpack, build, and run all tests.
|
||||||
|
|
||||||
> Example: `./x test distcheck`
|
> Example: `./x test distcheck`
|
||||||
|
|
||||||
### Tool tests
|
### Tool tests
|
||||||
|
|
||||||
Packages that are included with Rust have all of their tests run as well.
|
Packages that are included with Rust have all of their tests run as well. This
|
||||||
This includes things such as cargo, clippy, rustfmt, miri, bootstrap
|
includes things such as cargo, clippy, rustfmt, miri, bootstrap (testing the
|
||||||
(testing the Rust build system itself), etc.
|
Rust build system itself), etc.
|
||||||
|
|
||||||
Most of the tools are located in the [`src/tools`] directory.
|
Most of the tools are located in the [`src/tools`] directory. To run the tool's
|
||||||
To run the tool's tests, just pass its path to `./x test`.
|
tests, just pass its path to `./x test`.
|
||||||
|
|
||||||
> Example: `./x test src/tools/cargo`
|
> Example: `./x test src/tools/cargo`
|
||||||
|
|
||||||
Usually these tools involve running `cargo test` within the tool's directory.
|
Usually these tools involve running `cargo test` within the tool's directory.
|
||||||
|
|
||||||
If you want to run only a specified set of tests, append `--test-args FILTER_NAME` to the command.
|
If you want to run only a specified set of tests, append `--test-args
|
||||||
|
FILTER_NAME` to the command.
|
||||||
|
|
||||||
> Example: `./x test src/tools/miri --test-args padding`
|
> Example: `./x test src/tools/miri --test-args padding`
|
||||||
|
|
||||||
In CI, some tools are allowed to fail.
|
In CI, some tools are allowed to fail. Failures send notifications to the
|
||||||
Failures send notifications to the corresponding teams, and is tracked on the [toolstate website].
|
corresponding teams, and is tracked on the [toolstate website]. More information
|
||||||
More information can be found in the [toolstate documentation].
|
can be found in the [toolstate documentation].
|
||||||
|
|
||||||
[`src/tools`]: https://github.com/rust-lang/rust/tree/master/src/tools/
|
[`src/tools`]: https://github.com/rust-lang/rust/tree/master/src/tools/
|
||||||
[toolstate documentation]: https://forge.rust-lang.org/infra/toolstate.html
|
[toolstate documentation]: https://forge.rust-lang.org/infra/toolstate.html
|
||||||
[toolstate website]: https://rust-lang-nursery.github.io/rust-toolstate/
|
[toolstate website]: https://rust-lang-nursery.github.io/rust-toolstate/
|
||||||
|
|
||||||
### Integration testing
|
### Ecosystem testing
|
||||||
|
|
||||||
Rust tests integration with real-world code to catch regressions and make
|
Rust tests integration with real-world code to catch regressions and make
|
||||||
informed decisions about the evolution of the language. There are several kinds
|
informed decisions about the evolution of the language. There are several kinds
|
||||||
of integration tests, including Crater. See the [Integration testing
|
of ecosystem tests, including Crater. See the [Ecosystem testing
|
||||||
chapter](integration.md) for more details.
|
chapter](ecosystem.md) for more details.
|
||||||
|
|
||||||
### Performance testing
|
### Performance testing
|
||||||
|
|
||||||
A separate infrastructure is used for testing and tracking performance of the compiler.
|
A separate infrastructure is used for testing and tracking performance of the
|
||||||
See the [Performance testing chapter](perf.md) for more details.
|
compiler. See the [Performance testing chapter](perf.md) for more details.
|
||||||
|
|
||||||
## Further reading
|
## Further reading
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -4,14 +4,15 @@
|
||||||
|
|
||||||
A lot of work is put into improving the performance of the compiler and
|
A lot of work is put into improving the performance of the compiler and
|
||||||
preventing performance regressions.
|
preventing performance regressions.
|
||||||
|
|
||||||
The [rustc-perf](https://github.com/rust-lang/rustc-perf) project provides
|
The [rustc-perf](https://github.com/rust-lang/rustc-perf) project provides
|
||||||
several services for testing and tracking performance.
|
several services for testing and tracking performance. It provides hosted
|
||||||
It provides hosted infrastructure for running benchmarks as a service.
|
infrastructure for running benchmarks as a service. At this time, only
|
||||||
At this time, only `x86_64-unknown-linux-gnu` builds are tracked.
|
`x86_64-unknown-linux-gnu` builds are tracked.
|
||||||
|
|
||||||
A "perf run" is used to compare the performance of the compiler in different
|
A "perf run" is used to compare the performance of the compiler in different
|
||||||
configurations for a large collection of popular crates.
|
configurations for a large collection of popular crates. Different
|
||||||
Different configurations include "fresh builds", builds with incremental compilation, etc.
|
configurations include "fresh builds", builds with incremental compilation, etc.
|
||||||
|
|
||||||
The result of a perf run is a comparison between two versions of the compiler
|
The result of a perf run is a comparison between two versions of the compiler
|
||||||
(by their commit hashes).
|
(by their commit hashes).
|
||||||
|
|
@ -24,30 +25,29 @@ Any changes are noted in a comment on the PR.
|
||||||
|
|
||||||
### Manual perf runs
|
### Manual perf runs
|
||||||
|
|
||||||
Additionally, performance tests can be ran before a PR is merged on an as-needed basis.
|
Additionally, performance tests can be ran before a PR is merged on an as-needed
|
||||||
You should request a perf run if your PR may affect performance, especially if
|
basis. You should request a perf run if your PR may affect performance,
|
||||||
it can affect performance adversely.
|
especially if it can affect performance adversely.
|
||||||
|
|
||||||
To evaluate the performance impact of a PR, write this comment on the PR:
|
To evaluate the performance impact of a PR, write this comment on the PR:
|
||||||
|
|
||||||
`@bors try @rust-timer queue`
|
`@bors try @rust-timer queue`
|
||||||
|
|
||||||
> **Note**: Only users authorized to do perf runs are allowed to post this comment.
|
> **Note**: Only users authorized to do perf runs are allowed to post this
|
||||||
> Teams that are allowed to use it are tracked in the [Teams
|
> comment. Teams that are allowed to use it are tracked in the [Teams
|
||||||
> repository](https://github.com/rust-lang/team) with the `perf = true` value
|
> repository](https://github.com/rust-lang/team) with the `perf = true` value in
|
||||||
> in the `[permissions]` section (and bors permissions are also required).
|
> the `[permissions]` section (and bors permissions are also required). If you
|
||||||
> If you are not on one of those teams, feel free to ask for someone to post
|
> are not on one of those teams, feel free to ask for someone to post it for you
|
||||||
> it for you (either on Zulip or ask the assigned reviewer).
|
> (either on Zulip or ask the assigned reviewer).
|
||||||
|
|
||||||
This will first tell bors to do a "try" build which do a full release build
|
This will first tell bors to do a "try" build which do a full release build for
|
||||||
for `x86_64-unknown-linux-gnu`.
|
`x86_64-unknown-linux-gnu`. After the build finishes, it will place it in the
|
||||||
After the build finishes, it will place it in the queue to run the performance
|
queue to run the performance suite against it. After the performance tests
|
||||||
suite against it.
|
finish, the bot will post a comment on the PR with a summary and a link to a
|
||||||
After the performance tests finish, the bot will post a comment on the PR with
|
full report.
|
||||||
a summary and a link to a full report.
|
|
||||||
|
|
||||||
If you want to do a perf run for an already built artifact (e.g. for a previous try
|
If you want to do a perf run for an already built artifact (e.g. for a previous
|
||||||
build that wasn't benchmarked yet), you can run this instead:
|
try build that wasn't benchmarked yet), you can run this instead:
|
||||||
|
|
||||||
`@rust-timer build <commit-sha>`
|
`@rust-timer build <commit-sha>`
|
||||||
|
|
||||||
|
|
@ -56,5 +56,6 @@ You cannot benchmark the same artifact twice though.
|
||||||
More information about the available perf bot commands can be found
|
More information about the available perf bot commands can be found
|
||||||
[here](https://perf.rust-lang.org/help.html).
|
[here](https://perf.rust-lang.org/help.html).
|
||||||
|
|
||||||
More details about the benchmarking process itself are available in the [perf collector
|
More details about the benchmarking process itself are available in the [perf
|
||||||
|
collector
|
||||||
documentation](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md).
|
documentation](https://github.com/rust-lang/rustc-perf/blob/master/collector/README.md).
|
||||||
|
|
|
||||||
|
|
@ -2,68 +2,84 @@
|
||||||
|
|
||||||
<!-- toc -->
|
<!-- toc -->
|
||||||
|
|
||||||
You can run the tests using `x`. The most basic command – which
|
You can run the entire test collection using `x`. But note that running the
|
||||||
you will almost never want to use! – is as follows:
|
*entire* test collection is almost never what you want to do during local
|
||||||
|
development because it takes a really long time. For local development, see the
|
||||||
|
subsection after on how to run a subset of tests.
|
||||||
|
|
||||||
|
<div class="warning">
|
||||||
|
Running plain `./x test` will build the stage 1 compiler and then run the whole
|
||||||
|
test suite. This not only include `tests/`, but also `library/`, `compiler/`,
|
||||||
|
`src/tools/` package tests and more.
|
||||||
|
|
||||||
|
You usually only want to run a subset of the test suites (or even a smaller set
|
||||||
|
of tests than that) which you expect will exercise your changes. PR CI exercises
|
||||||
|
a subset of test collections, and merge queue CI will exercise all of the test
|
||||||
|
collection.
|
||||||
|
</div>
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test
|
./x test
|
||||||
```
|
```
|
||||||
|
|
||||||
This will build the stage 1 compiler and then run the whole test
|
The test results are cached and previously successful tests are `ignored` during
|
||||||
suite. You probably don't want to do this very often, because it takes
|
testing. The stdout/stderr contents as well as a timestamp file for every test
|
||||||
a very long time, and anyway bors / GitHub Actions will do it for you.
|
can be found under `build/<target-triple>/test/` for the given
|
||||||
(Often, I will run this command in the background after opening a PR that
|
`<target-triple>`. To force-rerun a test (e.g. in case the test runner fails to
|
||||||
I think is done, but rarely otherwise. -nmatsakis)
|
notice a change) you can use the `--force-rerun` CLI option.
|
||||||
|
|
||||||
The test results are cached and previously successful tests are
|
> **Note on requirements of external dependencies**
|
||||||
`ignored` during testing. The stdout/stderr contents as well as a
|
>
|
||||||
timestamp file for every test can be found under `build/ARCH/test/`.
|
> Some test suites may require external dependecies. This is especially true of
|
||||||
To force-rerun a test (e.g. in case the test runner fails to notice a change)
|
> debuginfo tests. Some debuginfo tests require a Python-enabled gdb. You can
|
||||||
you can simply remove the timestamp file, or use the `--force-rerun` CLI
|
> test if your gdb install supports Python by using the `python` command from
|
||||||
option.
|
> within gdb. Once invoked you can type some Python code (e.g. `print("hi")`)
|
||||||
|
> followed by return and then `CTRL+D` to execute it. If you are building gdb
|
||||||
Note that some tests require a Python-enabled gdb. You can test if
|
> from source, you will need to configure with
|
||||||
your gdb install supports Python by using the `python` command from
|
> `--with-python=<path-to-python-binary>`.
|
||||||
within gdb. Once invoked you can type some Python code (e.g.
|
|
||||||
`print("hi")`) followed by return and then `CTRL+D` to execute it.
|
|
||||||
If you are building gdb from source, you will need to configure with
|
|
||||||
`--with-python=<path-to-python-binary>`.
|
|
||||||
|
|
||||||
## Running a subset of the test suites
|
## Running a subset of the test suites
|
||||||
|
|
||||||
When working on a specific PR, you will usually want to run a smaller
|
When working on a specific PR, you will usually want to run a smaller set of
|
||||||
set of tests. For example, a good "smoke test" that can be used after
|
tests. For example, a good "smoke test" that can be used after modifying rustc
|
||||||
modifying rustc to see if things are generally working correctly would be the
|
to see if things are generally working correctly would be to exercise the `ui`
|
||||||
following:
|
test suite ([`tests/ui`]):
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test tests/ui
|
./x test tests/ui
|
||||||
```
|
```
|
||||||
|
|
||||||
This will run the `ui` test suite. Of course, the choice
|
This will run the `ui` test suite. Of course, the choice of test suites is
|
||||||
of test suites is somewhat arbitrary, and may not suit the task you are
|
somewhat arbitrary, and may not suit the task you are doing. For example, if you
|
||||||
doing. For example, if you are hacking on debuginfo, you may be better off
|
are hacking on debuginfo, you may be better off with the debuginfo test suite:
|
||||||
with the debuginfo test suite:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test tests/debuginfo
|
./x test tests/debuginfo
|
||||||
```
|
```
|
||||||
|
|
||||||
If you only need to test a specific subdirectory of tests for any
|
If you only need to test a specific subdirectory of tests for any given test
|
||||||
given test suite, you can pass that directory to `./x test`:
|
suite, you can pass that directory as a filter to `./x test`:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test tests/ui/const-generics
|
./x test tests/ui/const-generics
|
||||||
```
|
```
|
||||||
|
|
||||||
|
> **Note for MSYS2**
|
||||||
|
>
|
||||||
|
> On MSYS2 the paths seem to be strange and `./x test` neither recognizes
|
||||||
|
> `tests/ui/const-generics` nor `tests\ui\const-generics`. In that case, you can
|
||||||
|
> workaround it by using e.g. `./x test ui
|
||||||
|
> --test-args="tests/ui/const-generics"`.
|
||||||
|
|
||||||
Likewise, you can test a single file by passing its path:
|
Likewise, you can test a single file by passing its path:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test tests/ui/const-generics/const-test.rs
|
./x test tests/ui/const-generics/const-test.rs
|
||||||
```
|
```
|
||||||
|
|
||||||
`x` doesn't support running a single tool test by passing its path yet.
|
`x` doesn't support running a single tool test by passing its path yet. You'll
|
||||||
You'll have to use the `--test-args` argument as describled [below](#running-an-individual-test).
|
have to use the `--test-args` argument as describled
|
||||||
|
[below](#running-an-individual-test).
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test src/tools/miri --test-args tests/fail/uninit/padding-enum.rs
|
./x test src/tools/miri --test-args tests/fail/uninit/padding-enum.rs
|
||||||
|
|
@ -81,8 +97,8 @@ You'll have to use the `--test-args` argument as describled [below](#running-an-
|
||||||
./x test --stage 0 library/std
|
./x test --stage 0 library/std
|
||||||
```
|
```
|
||||||
|
|
||||||
Note that this only runs tests on `std`; if you want to test `core` or other crates,
|
Note that this only runs tests on `std`; if you want to test `core` or other
|
||||||
you have to specify those explicitly.
|
crates, you have to specify those explicitly.
|
||||||
|
|
||||||
### Run the tidy script and tests on the standard library
|
### Run the tidy script and tests on the standard library
|
||||||
|
|
||||||
|
|
@ -96,19 +112,23 @@ you have to specify those explicitly.
|
||||||
./x test --stage 1 library/std
|
./x test --stage 1 library/std
|
||||||
```
|
```
|
||||||
|
|
||||||
By listing which test suites you want to run you avoid having to run
|
By listing which test suites you want to run you avoid having to run tests for
|
||||||
tests for components you did not change at all.
|
components you did not change at all.
|
||||||
|
|
||||||
**Warning:** Note that bors only runs the tests with the full stage 2
|
<div class="warning">
|
||||||
build; therefore, while the tests **usually** work fine with stage 1,
|
Note that bors only runs the tests with the full stage 2 build; therefore, while
|
||||||
there are some limitations.
|
the tests **usually** work fine with stage 1, there are some limitations.
|
||||||
|
</div>
|
||||||
|
|
||||||
### Run all tests using a stage 2 compiler
|
### Run all tests using a stage 2 compiler
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test --stage 2
|
./x test --stage 2
|
||||||
```
|
```
|
||||||
|
|
||||||
|
<div class="warning">
|
||||||
You almost never need to do this; CI will run these tests for you.
|
You almost never need to do this; CI will run these tests for you.
|
||||||
|
</div>
|
||||||
|
|
||||||
## Run unit tests on the compiler/library
|
## Run unit tests on the compiler/library
|
||||||
|
|
||||||
|
|
@ -126,19 +146,19 @@ But unfortunately, it's impossible. You should invoke the following instead:
|
||||||
|
|
||||||
## Running an individual test
|
## Running an individual test
|
||||||
|
|
||||||
Another common thing that people want to do is to run an **individual
|
Another common thing that people want to do is to run an **individual test**,
|
||||||
test**, often the test they are trying to fix. As mentioned earlier,
|
often the test they are trying to fix. As mentioned earlier, you may pass the
|
||||||
you may pass the full file path to achieve this, or alternatively one
|
full file path to achieve this, or alternatively one may invoke `x` with the
|
||||||
may invoke `x` with the `--test-args` option:
|
`--test-args` option:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test tests/ui --test-args issue-1234
|
./x test tests/ui --test-args issue-1234
|
||||||
```
|
```
|
||||||
|
|
||||||
Under the hood, the test runner invokes the standard Rust test runner
|
Under the hood, the test runner invokes the standard Rust test runner (the same
|
||||||
(the same one you get with `#[test]`), so this command would wind up
|
one you get with `#[test]`), so this command would wind up filtering for tests
|
||||||
filtering for tests that include "issue-1234" in the name. (Thus
|
that include "issue-1234" in the name. Thus, `--test-args` is a good way to run
|
||||||
`--test-args` is a good way to run a collection of related tests.)
|
a collection of related tests.
|
||||||
|
|
||||||
## Passing arguments to `rustc` when running tests
|
## Passing arguments to `rustc` when running tests
|
||||||
|
|
||||||
|
|
@ -151,9 +171,9 @@ additional arguments to the compiler when building the tests.
|
||||||
|
|
||||||
## Editing and updating the reference files
|
## Editing and updating the reference files
|
||||||
|
|
||||||
If you have changed the compiler's output intentionally, or you are
|
If you have changed the compiler's output intentionally, or you are making a new
|
||||||
making a new test, you can pass `--bless` to the test subcommand. E.g.
|
test, you can pass `--bless` to the test subcommand. E.g. if some tests in
|
||||||
if some tests in `tests/ui` are failing, you can run
|
`tests/ui` are failing, you can run
|
||||||
|
|
||||||
```text
|
```text
|
||||||
./x test tests/ui --bless
|
./x test tests/ui --bless
|
||||||
|
|
@ -167,37 +187,35 @@ all tests. Of course you can also target just specific tests with the
|
||||||
|
|
||||||
There are a few options for running tests:
|
There are a few options for running tests:
|
||||||
|
|
||||||
* `config.toml` has the `rust.verbose-tests` option.
|
* `config.toml` has the `rust.verbose-tests` option. If `false`, each test will
|
||||||
If `false`, each test will print a single dot (the default).
|
print a single dot (the default). If `true`, the name of every test will be
|
||||||
If `true`, the name of every test will be printed.
|
printed. This is equivalent to the `--quiet` option in the [Rust test
|
||||||
This is equivalent to the `--quiet` option in the [Rust test
|
|
||||||
harness](https://doc.rust-lang.org/rustc/tests/)
|
harness](https://doc.rust-lang.org/rustc/tests/)
|
||||||
* The environment variable `RUST_TEST_THREADS` can be set to the number of
|
* The environment variable `RUST_TEST_THREADS` can be set to the number of
|
||||||
concurrent threads to use for testing.
|
concurrent threads to use for testing.
|
||||||
|
|
||||||
## Passing `--pass $mode`
|
## Passing `--pass $mode`
|
||||||
|
|
||||||
Pass UI tests now have three modes, `check-pass`, `build-pass` and
|
Pass UI tests now have three modes, `check-pass`, `build-pass` and `run-pass`.
|
||||||
`run-pass`. When `--pass $mode` is passed, these tests will be forced
|
When `--pass $mode` is passed, these tests will be forced to run under the given
|
||||||
to run under the given `$mode` unless the directive `// ignore-pass`
|
`$mode` unless the directive `//@ ignore-pass` exists in the test file. For
|
||||||
exists in the test file. For example, you can run all the tests in
|
example, you can run all the tests in `tests/ui` as `check-pass`:
|
||||||
`tests/ui` as `check-pass`:
|
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test tests/ui --pass check
|
./x test tests/ui --pass check
|
||||||
```
|
```
|
||||||
|
|
||||||
By passing `--pass $mode`, you can reduce the testing time. For each
|
By passing `--pass $mode`, you can reduce the testing time. For each mode,
|
||||||
mode, please see [Controlling pass/fail
|
please see [Controlling pass/fail
|
||||||
expectations](ui.md#controlling-passfail-expectations).
|
expectations](ui.md#controlling-passfail-expectations).
|
||||||
|
|
||||||
## Running tests with different "compare modes"
|
## Running tests with different "compare modes"
|
||||||
|
|
||||||
UI tests may have different output depending on certain "modes" that
|
UI tests may have different output depending on certain "modes" that the
|
||||||
the compiler is in. For example, when using the Polonius
|
compiler is in. For example, when using the Polonius mode, a test `foo.rs` will
|
||||||
mode, a test `foo.rs` will first look for expected output in
|
first look for expected output in `foo.polonius.stderr`, falling back to the
|
||||||
`foo.polonius.stderr`, falling back to the usual `foo.stderr` if not found.
|
usual `foo.stderr` if not found. The following will run the UI test suite in
|
||||||
The following will run the UI test suite in Polonius mode:
|
Polonius mode:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
./x test tests/ui --compare-mode=polonius
|
./x test tests/ui --compare-mode=polonius
|
||||||
|
|
@ -207,25 +225,25 @@ See [Compare modes](compiletest.md#compare-modes) for more details.
|
||||||
|
|
||||||
## Running tests manually
|
## Running tests manually
|
||||||
|
|
||||||
Sometimes it's easier and faster to just run the test by hand.
|
Sometimes it's easier and faster to just run the test by hand. Most tests are
|
||||||
Most tests are just `rs` files, so after
|
just `.rs` files, so after [creating a rustup
|
||||||
[creating a rustup toolchain](../building/how-to-build-and-run.md#creating-a-rustup-toolchain),
|
toolchain](../building/how-to-build-and-run.md#creating-a-rustup-toolchain), you
|
||||||
you can do something like:
|
can do something like:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
rustc +stage1 tests/ui/issue-1234.rs
|
rustc +stage1 tests/ui/issue-1234.rs
|
||||||
```
|
```
|
||||||
|
|
||||||
This is much faster, but doesn't always work. For example, some tests
|
This is much faster, but doesn't always work. For example, some tests include
|
||||||
include directives that specify specific compiler flags, or which rely
|
directives that specify specific compiler flags, or which rely on other crates,
|
||||||
on other crates, and they may not run the same without those options.
|
and they may not run the same without those options.
|
||||||
|
|
||||||
## Running `run-make` tests
|
## Running `run-make` tests
|
||||||
|
|
||||||
### Windows
|
### Windows
|
||||||
|
|
||||||
Running the `run-make` test suite on Windows is a bit more involved. There are numerous
|
Running the `run-make` test suite on Windows is a currently bit more involved.
|
||||||
prerequisites and environmental requirements:
|
There are numerous prerequisites and environmental requirements:
|
||||||
|
|
||||||
- Install msys2: <https://www.msys2.org/>
|
- Install msys2: <https://www.msys2.org/>
|
||||||
- Specify `MSYS2_PATH_TYPE=inherit` in `msys2.ini` in the msys2 installation directory, run the
|
- Specify `MSYS2_PATH_TYPE=inherit` in `msys2.ini` in the msys2 installation directory, run the
|
||||||
|
|
@ -236,29 +254,38 @@ prerequisites and environmental requirements:
|
||||||
- `pacman -S binutils`
|
- `pacman -S binutils`
|
||||||
- `./x test run-make` (`./x test tests/run-make` doesn't work)
|
- `./x test run-make` (`./x test tests/run-make` doesn't work)
|
||||||
|
|
||||||
|
There is [on-going work][port-run-make] to not rely on `Makefile`s in the
|
||||||
|
run-make test suite. Once this work is completed, you can run the entire
|
||||||
|
`run-make` test suite on native Windows inside `cmd` or `PowerShell` without
|
||||||
|
needing to install and use MSYS2. As of <!--date-check --> Oct 2024, it is
|
||||||
|
already possible to run the vast majority of the `run-make` test suite outside
|
||||||
|
of MSYS2, but there will be failures for the tests that still use `Makefile`s
|
||||||
|
due to not finding `make`.
|
||||||
|
|
||||||
## Running tests on a remote machine
|
## Running tests on a remote machine
|
||||||
|
|
||||||
Tests may be run on a remote machine (e.g. to test builds for a different
|
Tests may be run on a remote machine (e.g. to test builds for a different
|
||||||
architecture). This is done using `remote-test-client` on the build machine
|
architecture). This is done using `remote-test-client` on the build machine to
|
||||||
to send test programs to `remote-test-server` running on the remote machine.
|
send test programs to `remote-test-server` running on the remote machine.
|
||||||
`remote-test-server` executes the test programs and sends the results back to
|
`remote-test-server` executes the test programs and sends the results back to
|
||||||
the build machine. `remote-test-server` provides *unauthenticated remote code
|
the build machine. `remote-test-server` provides *unauthenticated remote code
|
||||||
execution* so be careful where it is used.
|
execution* so be careful where it is used.
|
||||||
|
|
||||||
To do this, first build `remote-test-server` for the remote
|
To do this, first build `remote-test-server` for the remote machine, e.g. for
|
||||||
machine, e.g. for RISC-V
|
RISC-V
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
./x build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
|
./x build src/tools/remote-test-server --target riscv64gc-unknown-linux-gnu
|
||||||
```
|
```
|
||||||
|
|
||||||
The binary will be created at
|
The binary will be created at
|
||||||
`./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy
|
`./build/host/stage2-tools/$TARGET_ARCH/release/remote-test-server`. Copy this
|
||||||
this over to the remote machine.
|
over to the remote machine.
|
||||||
|
|
||||||
On the remote machine, run the `remote-test-server` with the `--bind
|
On the remote machine, run the `remote-test-server` with the `--bind
|
||||||
0.0.0.0:12345` flag (and optionally `-v` for verbose output). Output should
|
0.0.0.0:12345` flag (and optionally `-v` for verbose output). Output should look
|
||||||
look like this:
|
like this:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
$ ./remote-test-server -v --bind 0.0.0.0:12345
|
$ ./remote-test-server -v --bind 0.0.0.0:12345
|
||||||
starting test server
|
starting test server
|
||||||
|
|
@ -272,6 +299,7 @@ restrictive IP address when binding.
|
||||||
|
|
||||||
You can test if the `remote-test-server` is working by connecting to it and
|
You can test if the `remote-test-server` is working by connecting to it and
|
||||||
sending `ping\n`. It should reply `pong`:
|
sending `ping\n`. It should reply `pong`:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
$ nc $REMOTE_IP 12345
|
$ nc $REMOTE_IP 12345
|
||||||
ping
|
ping
|
||||||
|
|
@ -281,13 +309,15 @@ pong
|
||||||
To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
|
To run tests using the remote runner, set the `TEST_DEVICE_ADDR` environment
|
||||||
variable then use `x` as usual. For example, to run `ui` tests for a RISC-V
|
variable then use `x` as usual. For example, to run `ui` tests for a RISC-V
|
||||||
machine with the IP address `1.2.3.4` use
|
machine with the IP address `1.2.3.4` use
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
export TEST_DEVICE_ADDR="1.2.3.4:12345"
|
export TEST_DEVICE_ADDR="1.2.3.4:12345"
|
||||||
./x test tests/ui --target riscv64gc-unknown-linux-gnu
|
./x test tests/ui --target riscv64gc-unknown-linux-gnu
|
||||||
```
|
```
|
||||||
|
|
||||||
If `remote-test-server` was run with the verbose flag, output on the test machine
|
If `remote-test-server` was run with the verbose flag, output on the test
|
||||||
may look something like
|
machine may look something like
|
||||||
|
|
||||||
```
|
```
|
||||||
[...]
|
[...]
|
||||||
run "/tmp/work/test1007/a"
|
run "/tmp/work/test1007/a"
|
||||||
|
|
@ -311,31 +341,28 @@ output) may fail without ever running on the remote machine.
|
||||||
|
|
||||||
## Testing on emulators
|
## Testing on emulators
|
||||||
|
|
||||||
Some platforms are tested via an emulator for architectures that aren't
|
Some platforms are tested via an emulator for architectures that aren't readily
|
||||||
readily available. For architectures where the standard library is well
|
available. For architectures where the standard library is well supported and
|
||||||
supported and the host operating system supports TCP/IP networking, see the
|
the host operating system supports TCP/IP networking, see the above instructions
|
||||||
above instructions for testing on a remote machine (in this case the
|
for testing on a remote machine (in this case the remote machine is emulated).
|
||||||
remote machine is emulated).
|
|
||||||
|
|
||||||
There is also a set of tools for orchestrating running the
|
There is also a set of tools for orchestrating running the tests within the
|
||||||
tests within the emulator. Platforms such as `arm-android` and
|
emulator. Platforms such as `arm-android` and `arm-unknown-linux-gnueabihf` are
|
||||||
`arm-unknown-linux-gnueabihf` are set up to automatically run the tests under
|
set up to automatically run the tests under emulation on GitHub Actions. The
|
||||||
emulation on GitHub Actions. The following will take a look at how a target's tests
|
following will take a look at how a target's tests are run under emulation.
|
||||||
are run under emulation.
|
|
||||||
|
|
||||||
The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
|
The Docker image for [armhf-gnu] includes [QEMU] to emulate the ARM CPU
|
||||||
architecture. Included in the Rust tree are the tools [remote-test-client]
|
architecture. Included in the Rust tree are the tools [remote-test-client] and
|
||||||
and [remote-test-server] which are programs for sending test programs and
|
[remote-test-server] which are programs for sending test programs and libraries
|
||||||
libraries to the emulator, and running the tests within the emulator, and
|
to the emulator, and running the tests within the emulator, and reading the
|
||||||
reading the results. The Docker image is set up to launch
|
results. The Docker image is set up to launch `remote-test-server` and the
|
||||||
`remote-test-server` and the build tools use `remote-test-client` to
|
build tools use `remote-test-client` to communicate with the server to
|
||||||
communicate with the server to coordinate running tests (see
|
coordinate running tests (see [src/bootstrap/src/core/build_steps/test.rs]).
|
||||||
[src/bootstrap/src/core/build_steps/test.rs]).
|
|
||||||
|
|
||||||
> TODO:
|
> **TODO**
|
||||||
> Is there any support for using an iOS emulator?
|
|
||||||
>
|
>
|
||||||
> It's also unclear to me how the wasm or asm.js tests are run.
|
> - Is there any support for using an iOS emulator?
|
||||||
|
> - It's also unclear to me how the wasm or asm.js tests are run.
|
||||||
|
|
||||||
[armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/host-x86_64/armhf-gnu/Dockerfile
|
[armhf-gnu]: https://github.com/rust-lang/rust/tree/master/src/ci/docker/host-x86_64/armhf-gnu/Dockerfile
|
||||||
[QEMU]: https://www.qemu.org/
|
[QEMU]: https://www.qemu.org/
|
||||||
|
|
@ -374,5 +401,9 @@ need to pass the library file path with `LIBRARY_PATH`:
|
||||||
$ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/12/ ./x test compiler/rustc_codegen_gcc/
|
$ LIBRARY_PATH=/usr/lib/gcc/x86_64-linux-gnu/12/ ./x test compiler/rustc_codegen_gcc/
|
||||||
```
|
```
|
||||||
|
|
||||||
If you encounter bugs or problems, don't hesitate to open issues on
|
If you encounter bugs or problems, don't hesitate to open issues on the
|
||||||
[rustc_codegen_gcc repository](https://github.com/rust-lang/rustc_codegen_gcc/).
|
[`rustc_codegen_gcc`
|
||||||
|
repository](https://github.com/rust-lang/rustc_codegen_gcc/).
|
||||||
|
|
||||||
|
[`tests/ui`]: https://github.com/rust-lang/rust/tree/master/tests/ui
|
||||||
|
[port-run-make]: https://github.com/rust-lang/rust/issues/121876
|
||||||
|
|
|
||||||
|
|
@ -1,32 +1,45 @@
|
||||||
# Rust for Linux integration tests
|
# Rust for Linux integration tests
|
||||||
|
|
||||||
[Rust for Linux](https://rust-for-linux.com/) (RfL) is an effort for adding support for the Rust programming
|
[Rust for Linux](https://rust-for-linux.com/) (RfL) is an effort for adding
|
||||||
language into the Linux kernel.
|
support for the Rust programming language into the Linux kernel.
|
||||||
|
|
||||||
## Building Rust for Linux in CI
|
## Building Rust for Linux in CI
|
||||||
|
|
||||||
Rust for Linux builds as part of the suite of bors tests that run before a pull request
|
Rust for Linux builds as part of the suite of bors tests that run before a pull
|
||||||
is merged.
|
request is merged.
|
||||||
|
|
||||||
The workflow builds a stage1 sysroot of the Rust compiler, downloads the Linux kernel, and tries to compile several Rust for Linux drivers and examples using this sysroot. RfL uses several unstable compiler/language features, therefore this workflow notifies us if a given compiler change would break it.
|
The workflow builds a stage1 sysroot of the Rust compiler, downloads the Linux
|
||||||
|
kernel, and tries to compile several Rust for Linux drivers and examples using
|
||||||
|
this sysroot. RfL uses several unstable compiler/language features, therefore
|
||||||
|
this workflow notifies us if a given compiler change would break it.
|
||||||
|
|
||||||
If you are worried that a pull request might break the Rust for Linux builder and want
|
If you are worried that a pull request might break the Rust for Linux builder
|
||||||
to test it out before submitting it to the bors queue, simply add this line to
|
and want to test it out before submitting it to the bors queue, simply add this
|
||||||
your PR description:
|
line to your PR description:
|
||||||
|
|
||||||
> try-job: x86_64-rust-for-linux
|
> try-job: x86_64-rust-for-linux
|
||||||
|
|
||||||
Then when you `@bors try` it will pick the job that builds the Rust for Linux integration.
|
Then when you `@bors try` it will pick the job that builds the Rust for Linux
|
||||||
|
integration.
|
||||||
|
|
||||||
## What to do in case of failure
|
## What to do in case of failure
|
||||||
|
|
||||||
Currently, we use the following unofficial policy for handling failures caused by a change breaking the RfL integration:
|
Currently, we use the following unofficial policy for handling failures caused
|
||||||
|
by a change breaking the RfL integration:
|
||||||
|
|
||||||
- If the breakage was unintentional, then fix the PR.
|
- If the breakage was unintentional, then fix the PR.
|
||||||
- If the breakage was intentional, then let [RFL][rfl-ping] know and discuss what will the kernel need to change.
|
- If the breakage was intentional, then let [RFL][rfl-ping] know and discuss
|
||||||
|
what will the kernel need to change.
|
||||||
- If the PR is urgent, then disable the test temporarily.
|
- If the PR is urgent, then disable the test temporarily.
|
||||||
- If the PR can wait a few days, then wait for RFL maintainers to provide a new Linux kernel commit hash with the needed changes done, and apply it to the PR, which would confirm the changes work.
|
- If the PR can wait a few days, then wait for RFL maintainers to provide a
|
||||||
|
new Linux kernel commit hash with the needed changes done, and apply it to
|
||||||
|
the PR, which would confirm the changes work.
|
||||||
|
|
||||||
If something goes wrong with the workflow, you can ping the [Rust for Linux][rfl-ping] ping group to ask for help.
|
If something goes wrong with the workflow, you can ping the [Rust for
|
||||||
|
Linux][rfl-ping] ping group to ask for help.
|
||||||
|
|
||||||
|
```text
|
||||||
|
@rustbot ping rfl
|
||||||
|
```
|
||||||
|
|
||||||
[rfl-ping]: ../notification-groups/rust-for-linux.md
|
[rfl-ping]: ../notification-groups/rust-for-linux.md
|
||||||
|
|
|
||||||
|
|
@ -1,36 +1,39 @@
|
||||||
# Suggest tests tool
|
# Suggest tests tool
|
||||||
|
|
||||||
This chapter is about the internals of and contribution instructions for the
|
This chapter is about the internals of and contribution instructions for the
|
||||||
`suggest-tests` tool. For a high-level overview of the tool, see
|
`suggest-tests` tool. For a high-level overview of the tool, see [this
|
||||||
[this section](../building/suggested.md#x-suggest). This tool is currently in a
|
section](../building/suggested.md#x-suggest). This tool is currently in a beta
|
||||||
beta state and is tracked by [this](https://github.com/rust-lang/rust/issues/109933)
|
state and is tracked by [this](https://github.com/rust-lang/rust/issues/109933)
|
||||||
issue on Github. Currently the number of tests it will suggest are very limited
|
issue on Github. Currently the number of tests it will suggest are very limited
|
||||||
in scope, we are looking to expand this (contributions welcome!).
|
in scope, we are looking to expand this (contributions welcome!).
|
||||||
|
|
||||||
## Internals
|
## Internals
|
||||||
|
|
||||||
The tool is defined in a separate crate ([`src/tools/suggest-tests`](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests))
|
The tool is defined in a separate crate
|
||||||
|
([`src/tools/suggest-tests`](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests))
|
||||||
which outputs suggestions which are parsed by a shim in bootstrap
|
which outputs suggestions which are parsed by a shim in bootstrap
|
||||||
([`src/bootstrap/src/core/build_steps/suggest.rs`](https://github.com/rust-lang/rust/blob/master/src/bootstrap/src/core/build_steps/suggest.rs)).
|
([`src/bootstrap/src/core/build_steps/suggest.rs`](https://github.com/rust-lang/rust/blob/master/src/bootstrap/src/core/build_steps/suggest.rs)).
|
||||||
The only notable thing the bootstrap shim does is (when invoked with the
|
The only notable thing the bootstrap shim does is (when invoked with the `--run`
|
||||||
`--run` flag) use bootstrap's internal mechanisms to create a new `Builder` and
|
flag) use bootstrap's internal mechanisms to create a new `Builder` and uses it
|
||||||
uses it to invoke the suggested commands. The `suggest-tests` crate is where the
|
to invoke the suggested commands. The `suggest-tests` crate is where the fun
|
||||||
fun happens, two kinds of suggestions are defined: "static" and "dynamic"
|
happens, two kinds of suggestions are defined: "static" and "dynamic"
|
||||||
suggestions.
|
suggestions.
|
||||||
|
|
||||||
### Static suggestions
|
### Static suggestions
|
||||||
|
|
||||||
Defined [here](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/static_suggestions.rs).
|
Defined
|
||||||
Static suggestions are simple: they are just [globs](https://crates.io/crates/glob)
|
[here](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/static_suggestions.rs).
|
||||||
which map to a `x` command. In `suggest-tests`, this is implemented with a
|
Static suggestions are simple: they are just
|
||||||
simple `macro_rules` macro.
|
[globs](https://crates.io/crates/glob) which map to a `x` command. In
|
||||||
|
`suggest-tests`, this is implemented with a simple `macro_rules` macro.
|
||||||
|
|
||||||
### Dynamic suggestions
|
### Dynamic suggestions
|
||||||
|
|
||||||
Defined [here](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/dynamic_suggestions.rs).
|
Defined
|
||||||
|
[here](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/dynamic_suggestions.rs).
|
||||||
These are more complicated than static suggestions and are implemented as
|
These are more complicated than static suggestions and are implemented as
|
||||||
functions with the following signature: `fn(&Path) -> Vec<Suggestion>`. In
|
functions with the following signature: `fn(&Path) -> Vec<Suggestion>`. In other
|
||||||
other words, each suggestion takes a path to a modified file and (after running
|
words, each suggestion takes a path to a modified file and (after running
|
||||||
arbitrary Rust code) can return any number of suggestions, or none. Dynamic
|
arbitrary Rust code) can return any number of suggestions, or none. Dynamic
|
||||||
suggestions are useful for situations where fine-grained control over
|
suggestions are useful for situations where fine-grained control over
|
||||||
suggestions is needed. For example, modifications to the `compiler/xyz/` path
|
suggestions is needed. For example, modifications to the `compiler/xyz/` path
|
||||||
|
|
@ -43,13 +46,14 @@ run.
|
||||||
The following steps should serve as a rough guide to add suggestions to
|
The following steps should serve as a rough guide to add suggestions to
|
||||||
`suggest-tests` (very welcome!):
|
`suggest-tests` (very welcome!):
|
||||||
|
|
||||||
1. Determine the rules for your suggestion. Is it simple and operates only on
|
1. Determine the rules for your suggestion. Is it simple and operates only on a
|
||||||
a single path or does it match globs? Does it need fine-grained control over
|
single path or does it match globs? Does it need fine-grained control over
|
||||||
the resulting command or does "one size fit all"?
|
the resulting command or does "one size fit all"?
|
||||||
2. Based on the previous step, decide if your suggestion should be implemented
|
2. Based on the previous step, decide if your suggestion should be implemented
|
||||||
as either static or dynamic.
|
as either static or dynamic.
|
||||||
3. Implement the suggestion. If it is dynamic then a test is highly recommended,
|
3. Implement the suggestion. If it is dynamic then a test is highly recommended,
|
||||||
to verify that your logic is correct and to give an example of the suggestion.
|
to verify that your logic is correct and to give an example of the
|
||||||
See the [tests.rs](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/tests.rs)
|
suggestion. See the
|
||||||
|
[tests.rs](https://github.com/rust-lang/rust/blob/master/src/tools/suggest-tests/src/tests.rs)
|
||||||
file.
|
file.
|
||||||
4. Open a PR implementing your suggestion. **(TODO: add example PR)**
|
4. Open a PR implementing your suggestion. **(TODO: add example PR)**
|
||||||
|
|
|
||||||
482
src/tests/ui.md
482
src/tests/ui.md
|
|
@ -2,105 +2,98 @@
|
||||||
|
|
||||||
<!-- toc -->
|
<!-- toc -->
|
||||||
|
|
||||||
UI tests are a particular [test suite](compiletest.md#test-suites) of compiletest.
|
UI tests are a particular [test suite](compiletest.md#test-suites) of
|
||||||
|
compiletest.
|
||||||
|
|
||||||
## Introduction
|
## Introduction
|
||||||
|
|
||||||
The tests in [`tests/ui`] are a collection of general-purpose tests which
|
The tests in [`tests/ui`] are a collection of general-purpose tests which
|
||||||
primarily focus on validating the console output of the compiler, but can be
|
primarily focus on validating the console output of the compiler, but can be
|
||||||
used for many other purposes.
|
used for many other purposes. For example, tests can also be configured to [run
|
||||||
For example, tests can also be configured to [run the resulting
|
the resulting program](#controlling-passfail-expectations) to verify its
|
||||||
program](#controlling-passfail-expectations) to verify its behavior.
|
behavior.
|
||||||
|
|
||||||
[`tests/ui`]: https://github.com/rust-lang/rust/blob/master/tests/ui
|
[`tests/ui`]: https://github.com/rust-lang/rust/blob/master/tests/ui
|
||||||
|
|
||||||
## General structure of a test
|
## General structure of a test
|
||||||
|
|
||||||
A test consists of a Rust source file located anywhere in the `tests/ui` directory.
|
A test consists of a Rust source file located anywhere in the `tests/ui`
|
||||||
For example, [`tests/ui/hello.rs`] is a basic hello-world test.
|
directory, but they should be placed in a suitable sub-directory. For example,
|
||||||
|
[`tests/ui/hello.rs`] is a basic hello-world test.
|
||||||
|
|
||||||
Compiletest will use `rustc` to compile the test, and compare the output
|
Compiletest will use `rustc` to compile the test, and compare the output against
|
||||||
against the expected output which is stored in a `.stdout` or `.stderr` file
|
the expected output which is stored in a `.stdout` or `.stderr` file located
|
||||||
located next to the test.
|
next to the test. See [Output comparison](#output-comparison) for more.
|
||||||
See [Output comparison](#output-comparison) for more.
|
|
||||||
|
|
||||||
Additionally, errors and warnings should be annotated with comments within
|
Additionally, errors and warnings should be annotated with comments within the
|
||||||
the source file.
|
source file. See [Error annotations](#error-annotations) for more.
|
||||||
See [Error annotations](#error-annotations) for more.
|
|
||||||
|
|
||||||
[Headers](headers.md) in the form of comments at the top of the file control
|
Compiletest [directives](directives.md) in the form of special comments prefixed
|
||||||
how the test is compiled and what the expected behavior is. Note that tests in
|
with `//@` control how the test is compiled and what the expected behavior is.
|
||||||
the "ui" test suite require the use of `//@ header-name` instead of
|
|
||||||
`// header-name` like the other test suites do. The other test suites will be
|
|
||||||
migrated to use the `//@` syntax too, but that is in progress. Additionally,
|
|
||||||
`// ignore-tidy` and `// ignore-tidy-*` are ignored by compiletest when
|
|
||||||
handling "ui" test suite tests (note that they are not `//@` directives).
|
|
||||||
|
|
||||||
Tests are expected to fail to compile, since most tests are testing compiler
|
Tests are expected to fail to compile, since most tests are testing compiler
|
||||||
errors.
|
errors. You can change that behavior with a directive, see [Controlling
|
||||||
You can change that behavior with a header, see [Controlling pass/fail
|
pass/fail expectations](#controlling-passfail-expectations).
|
||||||
expectations](#controlling-passfail-expectations).
|
|
||||||
|
|
||||||
By default, a test is built as an executable binary.
|
By default, a test is built as an executable binary. If you need a different
|
||||||
If you need a different crate type, you can use the `#![crate_type]` attribute
|
crate type, you can use the `#![crate_type]` attribute to set it as needed.
|
||||||
to set it as needed.
|
|
||||||
|
|
||||||
[`tests/ui/hello.rs`]: https://github.com/rust-lang/rust/blob/master/tests/ui/hello.rs
|
[`tests/ui/hello.rs`]: https://github.com/rust-lang/rust/blob/master/tests/ui/hello.rs
|
||||||
|
|
||||||
## Output comparison
|
## Output comparison
|
||||||
|
|
||||||
UI tests store the expected output from the compiler in `.stderr` and
|
UI tests store the expected output from the compiler in `.stderr` and `.stdout`
|
||||||
`.stdout` files next to the test.
|
snapshots next to the test. You normally generate these files with the `--bless`
|
||||||
You normally generate these files with the `--bless` CLI option, and then
|
CLI option, and then inspect them manually to verify they contain what you
|
||||||
inspect them manually to verify they contain what you expect.
|
expect.
|
||||||
|
|
||||||
The output is normalized to ignore unwanted differences, see the
|
The output is normalized to ignore unwanted differences, see the
|
||||||
[Normalization](#normalization) section.
|
[Normalization](#normalization) section. If the file is missing, then
|
||||||
If the file is missing, then compiletest expects the corresponding output to
|
compiletest expects the corresponding output to be empty.
|
||||||
be empty.
|
|
||||||
|
|
||||||
There can be multiple stdout/stderr files.
|
There can be multiple stdout/stderr files. The general form is:
|
||||||
The general form is:
|
|
||||||
|
|
||||||
|
```text
|
||||||
*test-name*`.`*revision*`.`*compare_mode*`.`*extension*
|
*test-name*`.`*revision*`.`*compare_mode*`.`*extension*
|
||||||
|
```
|
||||||
|
|
||||||
* *test-name* cannot contain dots. This is so that the general form of test
|
- *test-name* cannot contain dots. This is so that the general form of test
|
||||||
output filenames have a predictable form we can pattern match on in order to
|
output filenames have a predictable form we can pattern match on in order to
|
||||||
track stray test output files.
|
track stray test output files.
|
||||||
* *revision* is the [revision](#cfg-revisions) name.
|
- *revision* is the [revision](#cfg-revisions) name. This is not included when
|
||||||
This is not included when not using revisions.
|
not using revisions.
|
||||||
* *compare_mode* is the [compare mode](#compare-modes).
|
- *compare_mode* is the [compare mode](#compare-modes). This will only be
|
||||||
This will only be checked when the given compare mode is active.
|
checked when the given compare mode is active. If the file does not exist,
|
||||||
If the file does not exist, then compiletest will check for a file without
|
then compiletest will check for a file without the compare mode.
|
||||||
the compare mode.
|
- *extension* is the kind of output being checked:
|
||||||
* *extension* is the kind of output being checked:
|
- `stderr` — compiler stderr
|
||||||
* `stderr` — compiler stderr
|
- `stdout` — compiler stdout
|
||||||
* `stdout` — compiler stdout
|
- `run.stderr` — stderr when running the test
|
||||||
* `run.stderr` — stderr when running the test
|
- `run.stdout` — stdout when running the test
|
||||||
* `run.stdout` — stdout when running the test
|
- `64bit.stderr` — compiler stderr with `stderr-per-bitwidth` directive on a
|
||||||
* `64bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 64-bit target
|
64-bit target
|
||||||
* `32bit.stderr` — compiler stderr with `stderr-per-bitwidth` header on a 32-bit target
|
- `32bit.stderr` — compiler stderr with `stderr-per-bitwidth` directive on a
|
||||||
|
32-bit target
|
||||||
|
|
||||||
A simple example would be `foo.stderr` next to a `foo.rs` test.
|
A simple example would be `foo.stderr` next to a `foo.rs` test.
|
||||||
A more complex example would be `foo.my-revision.polonius.stderr`.
|
A more complex example would be `foo.my-revision.polonius.stderr`.
|
||||||
|
|
||||||
There are several [headers](headers.md) which will change how compiletest will
|
There are several [directives](directives.md) which will change how compiletest
|
||||||
check for output files:
|
will check for output files:
|
||||||
|
|
||||||
* `stderr-per-bitwidth` — checks separate output files based on the target
|
- `stderr-per-bitwidth` — checks separate output files based on the target
|
||||||
pointer width. Consider using the `normalize-stderr` header instead (see
|
pointer width. Consider using the `normalize-stderr` directive instead (see
|
||||||
[Normalization](#normalization)).
|
[Normalization](#normalization)).
|
||||||
* `dont-check-compiler-stderr` — Ignores stderr from the compiler.
|
- `dont-check-compiler-stderr` — Ignores stderr from the compiler.
|
||||||
* `dont-check-compiler-stdout` — Ignores stdout from the compiler.
|
- `dont-check-compiler-stdout` — Ignores stdout from the compiler.
|
||||||
* `compare-output-lines-by-subset` — Checks that the output contains the
|
- `compare-output-lines-by-subset` — Checks that the output contains the
|
||||||
contents of the stored output files by lines opposed to checking for strict
|
contents of the stored output files by lines opposed to checking for strict
|
||||||
equality.
|
equality.
|
||||||
|
|
||||||
UI tests run with `-Zdeduplicate-diagnostics=no` flag which disables
|
UI tests run with `-Zdeduplicate-diagnostics=no` flag which disables rustc's
|
||||||
rustc's built-in diagnostic deduplication mechanism.
|
built-in diagnostic deduplication mechanism. This means you may see some
|
||||||
This means you may see some duplicate messages in the output.
|
duplicate messages in the output. This helps illuminate situations where
|
||||||
This helps illuminate situations where duplicate diagnostics are being
|
duplicate diagnostics are being generated.
|
||||||
generated.
|
|
||||||
|
|
||||||
### Normalization
|
### Normalization
|
||||||
|
|
||||||
|
|
@ -109,20 +102,20 @@ platforms, mainly about filenames.
|
||||||
|
|
||||||
Compiletest makes the following replacements on the compiler output:
|
Compiletest makes the following replacements on the compiler output:
|
||||||
|
|
||||||
- The directory where the test is defined is replaced with `$DIR`.
|
- The directory where the test is defined is replaced with `$DIR`. Example:
|
||||||
Example: `/path/to/rust/tests/ui/error-codes`
|
`/path/to/rust/tests/ui/error-codes`
|
||||||
- The directory to the standard library source is replaced with `$SRC_DIR`.
|
- The directory to the standard library source is replaced with `$SRC_DIR`.
|
||||||
Example: `/path/to/rust/library`
|
Example: `/path/to/rust/library`
|
||||||
- Line and column numbers for paths in `$SRC_DIR` are replaced with `LL:COL`.
|
- Line and column numbers for paths in `$SRC_DIR` are replaced with `LL:COL`.
|
||||||
This helps ensure that changes to the layout of the standard library do not
|
This helps ensure that changes to the layout of the standard library do not
|
||||||
cause widespread changes to the `.stderr` files.
|
cause widespread changes to the `.stderr` files. Example:
|
||||||
Example: `$SRC_DIR/alloc/src/sync.rs:53:46`
|
`$SRC_DIR/alloc/src/sync.rs:53:46`
|
||||||
- The base directory where the test's output goes is replaced with `$TEST_BUILD_DIR`.
|
- The base directory where the test's output goes is replaced with
|
||||||
This only comes up in a few rare circumstances.
|
`$TEST_BUILD_DIR`. This only comes up in a few rare circumstances. Example:
|
||||||
Example: `/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui`
|
`/path/to/rust/build/x86_64-unknown-linux-gnu/test/ui`
|
||||||
- Tabs are replaced with `\t`.
|
- Tabs are replaced with `\t`.
|
||||||
- Backslashes (`\`) are converted to forward slashes (`/`) within paths (using
|
- Backslashes (`\`) are converted to forward slashes (`/`) within paths (using a
|
||||||
a heuristic). This helps normalize differences with Windows-style paths.
|
heuristic). This helps normalize differences with Windows-style paths.
|
||||||
- CRLF newlines are converted to LF.
|
- CRLF newlines are converted to LF.
|
||||||
- Error line annotations like `//~ ERROR some message` are removed.
|
- Error line annotations like `//~ ERROR some message` are removed.
|
||||||
- Various v0 and legacy symbol hashes are replaced with placeholders like
|
- Various v0 and legacy symbol hashes are replaced with placeholders like
|
||||||
|
|
@ -131,20 +124,20 @@ Compiletest makes the following replacements on the compiler output:
|
||||||
Additionally, the compiler is run with the `-Z ui-testing` flag which causes
|
Additionally, the compiler is run with the `-Z ui-testing` flag which causes
|
||||||
the compiler itself to apply some changes to the diagnostic output to make it
|
the compiler itself to apply some changes to the diagnostic output to make it
|
||||||
more suitable for UI testing.
|
more suitable for UI testing.
|
||||||
|
|
||||||
For example, it will anonymize line numbers in the output (line numbers
|
For example, it will anonymize line numbers in the output (line numbers
|
||||||
prefixing each source line are replaced with `LL`).
|
prefixing each source line are replaced with `LL`). In extremely rare
|
||||||
In extremely rare situations, this mode can be disabled with the header
|
situations, this mode can be disabled with the directive `//@
|
||||||
command `//@ compile-flags: -Z ui-testing=no`.
|
compile-flags: -Z ui-testing=no`.
|
||||||
|
|
||||||
Note: The line and column numbers for `-->` lines pointing to the test are
|
Note: The line and column numbers for `-->` lines pointing to the test are *not*
|
||||||
*not* normalized, and left as-is. This ensures that the compiler continues
|
normalized, and left as-is. This ensures that the compiler continues to point to
|
||||||
to point to the correct location, and keeps the stderr files readable.
|
the correct location, and keeps the stderr files readable. Ideally all
|
||||||
Ideally all line/column information would be retained, but small changes to
|
line/column information would be retained, but small changes to the source
|
||||||
the source causes large diffs, and more frequent merge conflicts and test
|
causes large diffs, and more frequent merge conflicts and test errors.
|
||||||
errors.
|
|
||||||
|
|
||||||
Sometimes these built-in normalizations are not enough. In such cases, you
|
Sometimes these built-in normalizations are not enough. In such cases, you may
|
||||||
may provide custom normalization rules using the header commands, e.g.
|
provide custom normalization rules using `normalize-*` directives, e.g.
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
//@ normalize-stdout-test: "foo" -> "bar"
|
//@ normalize-stdout-test: "foo" -> "bar"
|
||||||
|
|
@ -152,10 +145,10 @@ may provide custom normalization rules using the header commands, e.g.
|
||||||
//@ normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)"
|
//@ normalize-stderr-64bit: "fn\(\) \(64 bits\)" -> "fn\(\) \($$PTR bits\)"
|
||||||
```
|
```
|
||||||
|
|
||||||
This tells the test, on 32-bit platforms, whenever the compiler writes
|
This tells the test, on 32-bit platforms, whenever the compiler writes `fn() (32
|
||||||
`fn() (32 bits)` to stderr, it should be normalized to read `fn() ($PTR bits)`
|
bits)` to stderr, it should be normalized to read `fn() ($PTR bits)` instead.
|
||||||
instead. Similar for 64-bit. The replacement is performed by regexes using
|
Similar for 64-bit. The replacement is performed by regexes using default regex
|
||||||
default regex flavor provided by `regex` crate.
|
flavor provided by `regex` crate.
|
||||||
|
|
||||||
The corresponding reference file will use the normalized output to test both
|
The corresponding reference file will use the normalized output to test both
|
||||||
32-bit and 64-bit platforms:
|
32-bit and 64-bit platforms:
|
||||||
|
|
@ -168,22 +161,21 @@ The corresponding reference file will use the normalized output to test both
|
||||||
...
|
...
|
||||||
```
|
```
|
||||||
|
|
||||||
Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`] for a
|
Please see [`ui/transmute/main.rs`][mrs] and [`main.stderr`] for a concrete
|
||||||
concrete usage example.
|
usage example.
|
||||||
|
|
||||||
[mrs]: https://github.com/rust-lang/rust/blob/master/tests/ui/transmute/main.rs
|
[mrs]: https://github.com/rust-lang/rust/blob/master/tests/ui/transmute/main.rs
|
||||||
[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/tests/ui/transmute/main.stderr
|
[`main.stderr`]: https://github.com/rust-lang/rust/blob/master/tests/ui/transmute/main.stderr
|
||||||
|
|
||||||
Besides `normalize-stderr-32bit` and `-64bit`, one may use any target
|
Besides `normalize-stderr-32bit` and `-64bit`, one may use any target
|
||||||
information or stage supported by [`ignore-X`](headers.md#ignoring-tests)
|
information or stage supported by [`ignore-X`](directives.md#ignoring-tests) here
|
||||||
here as well (e.g. `normalize-stderr-windows` or simply
|
as well (e.g. `normalize-stderr-windows` or simply `normalize-stderr-test` for
|
||||||
`normalize-stderr-test` for unconditional replacement).
|
unconditional replacement).
|
||||||
|
|
||||||
|
|
||||||
## Error annotations
|
## Error annotations
|
||||||
|
|
||||||
Error annotations specify the errors that the compiler is expected to emit.
|
Error annotations specify the errors that the compiler is expected to emit. They
|
||||||
They are "attached" to the line in source where the error is located.
|
are "attached" to the line in source where the error is located.
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
fn main() {
|
fn main() {
|
||||||
|
|
@ -191,41 +183,46 @@ fn main() {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
Although UI tests have a `.stderr` file which contains the entire compiler output,
|
Although UI tests have a `.stderr` file which contains the entire compiler
|
||||||
UI tests require that errors are also annotated within the source.
|
output, UI tests require that errors are also annotated within the source. This
|
||||||
This redundancy helps avoid mistakes since the `.stderr` files are usually
|
redundancy helps avoid mistakes since the `.stderr` files are usually
|
||||||
auto-generated.
|
auto-generated. It also helps to directly see where the error spans are expected
|
||||||
It also helps to directly see where the error spans are expected to point to
|
to point to by looking at one file instead of having to compare the `.stderr`
|
||||||
by looking at one file instead of having to compare the `.stderr` file with
|
file with the source. Finally, they ensure that no additional unexpected errors
|
||||||
the source.
|
are generated.
|
||||||
Finally, they ensure that no additional unexpected errors are generated.
|
|
||||||
|
|
||||||
They have several forms, but generally are a comment with the diagnostic
|
They have several forms, but generally are a comment with the diagnostic level
|
||||||
level (such as `ERROR`) and a substring of the expected error output.
|
(such as `ERROR`) and a substring of the expected error output. You don't have
|
||||||
You don't have to write out the entire message, just make sure to include the
|
to write out the entire message, just make sure to include the important part of
|
||||||
important part of the message to make it self-documenting.
|
the message to make it self-documenting.
|
||||||
|
|
||||||
The error annotation needs to match with the line of the diagnostic.
|
The error annotation needs to match with the line of the diagnostic. There are
|
||||||
There are several ways to match the message with the line (see the examples below):
|
several ways to match the message with the line (see the examples below):
|
||||||
|
|
||||||
* `~`: Associates the error level and message with the current line
|
* `~`: Associates the error level and message with the *current* line
|
||||||
* `~^`: Associates the error level and message with the previous error
|
* `~^`: Associates the error level and message with the *previous* error
|
||||||
annotation line.
|
annotation line. Each caret (`^`) that you add adds a line to this, so `~^^^`
|
||||||
Each caret (`^`) that you add adds a line to this, so `~^^^` is three lines
|
is three lines above the error annotation line.
|
||||||
above the error annotation line.
|
* `~|`: Associates the error level and message with the *same* line as the
|
||||||
* `~|`: Associates the error level and message with the same line as the
|
*previous comment*. This is more convenient than using multiple carets when
|
||||||
previous comment.
|
there are multiple messages associated with the same line.
|
||||||
This is more convenient than using multiple carets when there are multiple
|
|
||||||
messages associated with the same line.
|
|
||||||
|
|
||||||
The space character between `//~` (or other variants) and the subsequent text
|
Example:
|
||||||
is negligible (i.e. there is no semantic difference between `//~ ERROR` and
|
|
||||||
|
```rust,ignore
|
||||||
|
let _ = same_line; //~ ERROR undeclared variable
|
||||||
|
fn meow(_: [u8]) {}
|
||||||
|
//~^ ERROR unsized
|
||||||
|
//~| ERROR anonymous parameters
|
||||||
|
```
|
||||||
|
|
||||||
|
The space character between `//~` (or other variants) and the subsequent text is
|
||||||
|
negligible (i.e. there is no semantic difference between `//~ ERROR` and
|
||||||
`//~ERROR` although the former is more common in the codebase).
|
`//~ERROR` although the former is more common in the codebase).
|
||||||
|
|
||||||
### Error annotation examples
|
### Error annotation examples
|
||||||
|
|
||||||
Here are examples of error annotations on different lines of UI test
|
Here are examples of error annotations on different lines of UI test source.
|
||||||
source.
|
|
||||||
|
|
||||||
#### Positioned on error line
|
#### Positioned on error line
|
||||||
|
|
||||||
|
|
@ -243,10 +240,9 @@ fn main() {
|
||||||
|
|
||||||
#### Positioned below error line
|
#### Positioned below error line
|
||||||
|
|
||||||
Use the `//~^` idiom with number of carets in the string to indicate the
|
Use the `//~^` idiom with number of carets in the string to indicate the number
|
||||||
number of lines above.
|
of lines above. In the example below, the error line is four lines above the
|
||||||
In the example below, the error line is four lines above the error annotation
|
error annotation line so four carets are included in the annotation.
|
||||||
line so four carets are included in the annotation.
|
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
fn main() {
|
fn main() {
|
||||||
|
|
@ -280,8 +276,8 @@ fn main() {
|
||||||
|
|
||||||
### `error-pattern`
|
### `error-pattern`
|
||||||
|
|
||||||
The `error-pattern` [header](headers.md) can be used for
|
The `error-pattern` [directive](directives.md) can be used for messages that don't
|
||||||
messages that don't have a specific span.
|
have a specific span.
|
||||||
|
|
||||||
Let's think about this test:
|
Let's think about this test:
|
||||||
|
|
||||||
|
|
@ -294,9 +290,9 @@ fn main() {
|
||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
We want to ensure this shows "index out of bounds" but we cannot use the
|
We want to ensure this shows "index out of bounds" but we cannot use the `ERROR`
|
||||||
`ERROR` annotation since the error doesn't have any span.
|
annotation since the error doesn't have any span. Then it's time to use the
|
||||||
Then it's time to use the `error-pattern` header:
|
`error-pattern` directive:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
//@ error-pattern: index out of bounds
|
//@ error-pattern: index out of bounds
|
||||||
|
|
@ -314,33 +310,32 @@ But for strict testing, try to use the `ERROR` annotation as much as possible.
|
||||||
|
|
||||||
The error levels that you can have are:
|
The error levels that you can have are:
|
||||||
|
|
||||||
1. `ERROR`
|
- `ERROR`
|
||||||
2. `WARN` or `WARNING`
|
- `WARN` or `WARNING`
|
||||||
3. `NOTE`
|
- `NOTE`
|
||||||
4. `HELP` and `SUGGESTION`
|
- `HELP` and `SUGGESTION`
|
||||||
|
|
||||||
You are allowed to not include a level, but you should include it at least for
|
You are allowed to not include a level, but you should include it at least for
|
||||||
the primary message.
|
the primary message.
|
||||||
|
|
||||||
The `SUGGESTION` level is used for specifying what the expected replacement
|
The `SUGGESTION` level is used for specifying what the expected replacement text
|
||||||
text should be for a diagnostic suggestion.
|
should be for a diagnostic suggestion.
|
||||||
|
|
||||||
UI tests use the `-A unused` flag by default to ignore all unused warnings, as
|
UI tests use the `-A unused` flag by default to ignore all unused warnings, as
|
||||||
unused warnings are usually not the focus of a test.
|
unused warnings are usually not the focus of a test. However, simple code
|
||||||
However, simple code samples often have unused warnings.
|
samples often have unused warnings. If the test is specifically testing an
|
||||||
If the test is specifically testing an unused warning, just add the
|
unused warning, just add the appropriate `#![warn(unused)]` attribute as needed.
|
||||||
appropriate `#![warn(unused)]` attribute as needed.
|
|
||||||
|
|
||||||
### cfg revisions
|
### `cfg` revisions
|
||||||
|
|
||||||
When using [revisions](compiletest.md#revisions), different messages can be
|
When using [revisions](compiletest.md#revisions), different messages can be
|
||||||
conditionally checked based on the current revision.
|
conditionally checked based on the current revision. This is done by placing the
|
||||||
This is done by placing the revision cfg name in brackets like this:
|
revision cfg name in brackets like this:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
//@ edition:2018
|
//@ edition:2018
|
||||||
//@ revisions: mir thir
|
//@ revisions: mir thir
|
||||||
//@ [thir]compile-flags: -Z thir-unsafeck
|
//@[thir] compile-flags: -Z thir-unsafeck
|
||||||
|
|
||||||
async unsafe fn f() {}
|
async unsafe fn f() {}
|
||||||
|
|
||||||
|
|
@ -356,10 +351,10 @@ fn main() {
|
||||||
In this example, the second error message is only emitted in the `mir` revision.
|
In this example, the second error message is only emitted in the `mir` revision.
|
||||||
The `thir` revision only emits the first error.
|
The `thir` revision only emits the first error.
|
||||||
|
|
||||||
If the cfg causes the compiler to emit different output, then a test can have
|
If the `cfg` causes the compiler to emit different output, then a test can have
|
||||||
multiple `.stderr` files for the different outputs.
|
multiple `.stderr` files for the different outputs. In the example above, there
|
||||||
In the example above, there would be a `.mir.stderr` and `.thir.stderr` file
|
would be a `.mir.stderr` and `.thir.stderr` file with the different outputs of
|
||||||
with the different outputs of the different revisions.
|
the different revisions.
|
||||||
|
|
||||||
> Note: cfg revisions also work inside the source code with `#[cfg]` attributes.
|
> Note: cfg revisions also work inside the source code with `#[cfg]` attributes.
|
||||||
>
|
>
|
||||||
|
|
@ -368,113 +363,108 @@ with the different outputs of the different revisions.
|
||||||
## Controlling pass/fail expectations
|
## Controlling pass/fail expectations
|
||||||
|
|
||||||
By default, a UI test is expected to **generate a compile error** because most
|
By default, a UI test is expected to **generate a compile error** because most
|
||||||
of the tests are checking for invalid input and error diagnostics.
|
of the tests are checking for invalid input and error diagnostics. However, you
|
||||||
However, you can also make UI tests where compilation is expected to succeed,
|
can also make UI tests where compilation is expected to succeed, and you can
|
||||||
and you can even run the resulting program.
|
even run the resulting program. Just add one of the following
|
||||||
Just add one of the following [header commands](headers.md):
|
[directives](directives.md):
|
||||||
|
|
||||||
* Pass headers:
|
- Pass directives:
|
||||||
* `//@ check-pass` — compilation should succeed but skip codegen
|
- `//@ check-pass` — compilation should succeed but skip codegen
|
||||||
(which is expensive and isn't supposed to fail in most cases).
|
(which is expensive and isn't supposed to fail in most cases).
|
||||||
* `//@ build-pass` — compilation and linking should succeed but do
|
- `//@ build-pass` — compilation and linking should succeed but do
|
||||||
not run the resulting binary.
|
not run the resulting binary.
|
||||||
* `//@ run-pass` — compilation should succeed and running the resulting
|
- `//@ run-pass` — compilation should succeed and running the resulting
|
||||||
binary should also succeed.
|
binary should also succeed.
|
||||||
* Fail headers:
|
- Fail directives:
|
||||||
* `//@ check-fail` — compilation should fail (the codegen phase is skipped).
|
- `//@ check-fail` — compilation should fail (the codegen phase is skipped).
|
||||||
This is the default for UI tests.
|
This is the default for UI tests.
|
||||||
* `//@ build-fail` — compilation should fail during the codegen phase.
|
- `//@ build-fail` — compilation should fail during the codegen phase.
|
||||||
This will run `rustc` twice, once to verify that it compiles successfully
|
This will run `rustc` twice, once to verify that it compiles successfully
|
||||||
without the codegen phase, then a second time the full compile should
|
without the codegen phase, then a second time the full compile should
|
||||||
fail.
|
fail.
|
||||||
* `//@ run-fail` — compilation should succeed, but running the resulting
|
- `//@ run-fail` — compilation should succeed, but running the resulting
|
||||||
binary should fail.
|
binary should fail.
|
||||||
|
|
||||||
For `run-pass` and `run-fail` tests, by default the output of the program
|
For `run-pass` and `run-fail` tests, by default the output of the program itself
|
||||||
itself is not checked.
|
is not checked.
|
||||||
If you want to check the output of running the program, include the
|
|
||||||
`check-run-results` header.
|
|
||||||
This will check for a `.run.stderr` and `.run.stdout` files to compare
|
|
||||||
against the actual output of the program.
|
|
||||||
|
|
||||||
Tests with the `*-pass` headers can be overridden with the `--pass`
|
If you want to check the output of running the program, include the
|
||||||
|
`check-run-results` directive. This will check for a `.run.stderr` and
|
||||||
|
`.run.stdout` files to compare against the actual output of the program.
|
||||||
|
|
||||||
|
Tests with the `*-pass` directives can be overridden with the `--pass`
|
||||||
command-line option:
|
command-line option:
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
./x test tests/ui --pass check
|
./x test tests/ui --pass check
|
||||||
```
|
```
|
||||||
|
|
||||||
The `--pass` option only affects UI tests.
|
The `--pass` option only affects UI tests. Using `--pass check` can run the UI
|
||||||
Using `--pass check` can run the UI test suite much faster (roughly twice as
|
test suite much faster (roughly twice as fast on my system), though obviously
|
||||||
fast on my system), though obviously not exercising as much.
|
not exercising as much.
|
||||||
|
|
||||||
The `ignore-pass` header can be used to ignore the `--pass` CLI flag if the
|
The `ignore-pass` directive can be used to ignore the `--pass` CLI flag if the
|
||||||
test won't work properly with that override.
|
test won't work properly with that override.
|
||||||
|
|
||||||
|
|
||||||
## Known bugs
|
## Known bugs
|
||||||
|
|
||||||
The `known-bug` header may be used for tests that demonstrate a known bug that
|
The `known-bug` directive may be used for tests that demonstrate a known bug
|
||||||
has not yet been fixed.
|
that has not yet been fixed. Adding tests for known bugs is helpful for several
|
||||||
Adding tests for known bugs is helpful for several reasons, including:
|
reasons, including:
|
||||||
|
|
||||||
1. Maintaining a functional test that can be conveniently reused when the bug is fixed.
|
1. Maintaining a functional test that can be conveniently reused when the bug is
|
||||||
2. Providing a sentinel that will fail if the bug is incidentally fixed.
|
fixed.
|
||||||
This can alert the developer so they know that the associated issue has
|
2. Providing a sentinel that will fail if the bug is incidentally fixed. This
|
||||||
been fixed and can possibly be closed.
|
can alert the developer so they know that the associated issue has been fixed
|
||||||
|
and can possibly be closed.
|
||||||
|
|
||||||
Do not include [error annotations](#error-annotations) in a test with `known-bug`.
|
Do not include [error annotations](#error-annotations) in a test with
|
||||||
The test should still include other normal headers and stdout/stderr files.
|
`known-bug`. The test should still include other normal directives and
|
||||||
|
stdout/stderr files.
|
||||||
|
|
||||||
|
|
||||||
## Test organization
|
## Test organization
|
||||||
|
|
||||||
When deciding where to place a test file, please try to find a subdirectory
|
When deciding where to place a test file, please try to find a subdirectory that
|
||||||
that best matches what you are trying to exercise.
|
best matches what you are trying to exercise. Do your best to keep things
|
||||||
Do your best to keep things organized.
|
organized. Admittedly it can be difficult as some tests can overlap different
|
||||||
Admittedly it can be difficult as some tests can overlap different categories,
|
categories, and the existing layout may not fit well.
|
||||||
and the existing layout may not fit well.
|
|
||||||
|
|
||||||
For regression tests – basically, some random snippet of code that came in
|
Name the test by a concise description of what the test is checking. Avoid
|
||||||
from the internet – we often name the test after the issue plus a short
|
including the issue number in the test name. See [best
|
||||||
description.
|
practices](best-practices.md) for a more in-depth discussion of this.
|
||||||
Ideally, the test should be added to a directory that helps identify what
|
|
||||||
piece of code is being tested here (e.g.,
|
|
||||||
`tests/ui/borrowck/issue-54597-reject-move-out-of-borrow-via-pat.rs`)
|
|
||||||
|
|
||||||
When writing a new feature, **create a subdirectory to store your tests**.
|
Ideally, the test should be added to a directory that helps identify what piece
|
||||||
For example, if you are implementing RFC 1234 ("Widgets"), then it might make
|
of code is being tested here (e.g.,
|
||||||
sense to put the tests in a directory like `tests/ui/rfc1234-widgets/`.
|
`tests/ui/borrowck/reject-move-out-of-borrow-via-pat.rs`)
|
||||||
|
|
||||||
In other cases, there may already be a suitable directory. (The proper
|
When writing a new feature, you may want to **create a subdirectory to store
|
||||||
directory structure to use is actually an area of active debate.)
|
your tests**. For example, if you are implementing RFC 1234 ("Widgets"), then it
|
||||||
|
might make sense to put the tests in a directory like
|
||||||
|
`tests/ui/rfc1234-widgets/`.
|
||||||
|
|
||||||
Over time, the [`tests/ui`] directory has grown very fast.
|
In other cases, there may already be a suitable directory.
|
||||||
There is a check in [tidy](intro.md#tidy) that will ensure none of the
|
|
||||||
subdirectories has more than 1000 entries.
|
Over time, the [`tests/ui`] directory has grown very fast. There is a check in
|
||||||
Having too many files causes problems because it isn't editor/IDE friendly and
|
[tidy](intro.md#tidy) that will ensure none of the subdirectories has more than
|
||||||
the GitHub UI won't show more than 1000 entries.
|
1000 entries. Having too many files causes problems because it isn't editor/IDE
|
||||||
However, since `tests/ui` (UI test root directory) and `tests/ui/issues`
|
friendly and the GitHub UI won't show more than 1000 entries. However, since
|
||||||
directories have more than 1000 entries, we set a different limit for those
|
`tests/ui` (UI test root directory) and `tests/ui/issues` directories have more
|
||||||
directories.
|
than 1000 entries, we set a different limit for those directories. So, please
|
||||||
So, please avoid putting a new test there and try to find a more relevant
|
avoid putting a new test there and try to find a more relevant place.
|
||||||
place.
|
|
||||||
|
|
||||||
For example, if your test is related to closures, you should put it in
|
For example, if your test is related to closures, you should put it in
|
||||||
`tests/ui/closures`.
|
`tests/ui/closures`. When you reach the limit, you could increase it by tweaking
|
||||||
If you're not sure where is the best place, it's still okay to add to
|
[here][ui test tidy].
|
||||||
`tests/ui/issues/`.
|
|
||||||
When you reach the limit, you could increase it by tweaking [here][ui test
|
|
||||||
tidy].
|
|
||||||
|
|
||||||
[ui test tidy]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/ui_tests.rs
|
[ui test tidy]: https://github.com/rust-lang/rust/blob/master/src/tools/tidy/src/ui_tests.rs
|
||||||
|
|
||||||
|
|
||||||
## Rustfix tests
|
## Rustfix tests
|
||||||
|
|
||||||
UI tests can validate that diagnostic suggestions apply correctly
|
UI tests can validate that diagnostic suggestions apply correctly and that the
|
||||||
and that the resulting changes compile correctly.
|
resulting changes compile correctly. This can be done with the `run-rustfix`
|
||||||
This can be done with the `run-rustfix` header:
|
directive:
|
||||||
|
|
||||||
```rust,ignore
|
```rust,ignore
|
||||||
//@ run-rustfix
|
//@ run-rustfix
|
||||||
|
|
@ -487,40 +477,40 @@ pub struct not_camel_case {}
|
||||||
//~| SUGGESTION NotCamelCase
|
//~| SUGGESTION NotCamelCase
|
||||||
```
|
```
|
||||||
|
|
||||||
Rustfix tests should have a file with the `.fixed` extension which contains
|
Rustfix tests should have a file with the `.fixed` extension which contains the
|
||||||
the source file after the suggestion has been applied.
|
source file after the suggestion has been applied.
|
||||||
|
|
||||||
When the test is run, compiletest first checks that the correct
|
- When the test is run, compiletest first checks that the correct lint/warning
|
||||||
lint/warning is generated.
|
is generated.
|
||||||
Then, it applies the suggestion and compares against `.fixed` (they must match).
|
- Then, it applies the suggestion and compares against `.fixed` (they must
|
||||||
Finally, the fixed source is compiled, and this compilation is required to succeed.
|
match).
|
||||||
|
- Finally, the fixed source is compiled, and this compilation is required to
|
||||||
|
succeed.
|
||||||
|
|
||||||
Usually when creating a rustfix test you will generate the `.fixed` file
|
Usually when creating a rustfix test you will generate the `.fixed` file
|
||||||
automatically with the `x test --bless` option.
|
automatically with the `x test --bless` option.
|
||||||
|
|
||||||
The `run-rustfix` header will cause *all* suggestions to be applied, even
|
The `run-rustfix` directive will cause *all* suggestions to be applied, even if
|
||||||
if they are not [`MachineApplicable`](../diagnostics.md#suggestions).
|
they are not [`MachineApplicable`](../diagnostics.md#suggestions). If this is a
|
||||||
If this is a problem, then you can add the `rustfix-only-machine-applicable`
|
problem, then you can add the `rustfix-only-machine-applicable` directive in
|
||||||
header in addition to `run-rustfix`.
|
addition to `run-rustfix`. This should be used if there is a mixture of
|
||||||
This should be used if there is a mixture of different suggestion levels, and
|
different suggestion levels, and some of the non-machine-applicable ones do not
|
||||||
some of the non-machine-applicable ones do not apply cleanly.
|
apply cleanly.
|
||||||
|
|
||||||
|
|
||||||
## Compare modes
|
## Compare modes
|
||||||
|
|
||||||
[Compare modes](compiletest.md#compare-modes) can be used to run all tests
|
[Compare modes](compiletest.md#compare-modes) can be used to run all tests with
|
||||||
with different flags from what they are normally compiled with.
|
different flags from what they are normally compiled with. In some cases, this
|
||||||
In some cases, this might result in different output from the compiler.
|
might result in different output from the compiler. To support this, different
|
||||||
To support this, different output files can be saved which contain the
|
output files can be saved which contain the output based on the compare mode.
|
||||||
output based on the compare mode.
|
|
||||||
|
|
||||||
For example, when using the Polonius mode, a test `foo.rs` will
|
For example, when using the Polonius mode, a test `foo.rs` will first look for
|
||||||
first look for expected output in `foo.polonius.stderr`, falling back to the usual
|
expected output in `foo.polonius.stderr`, falling back to the usual `foo.stderr`
|
||||||
`foo.stderr` if not found.
|
if not found. This is useful as different modes can sometimes result in
|
||||||
This is useful as different modes can sometimes result in different
|
different diagnostics and behavior. This can help track which tests have
|
||||||
diagnostics and behavior.
|
differences between the modes, and to visually inspect those diagnostic
|
||||||
This can help track which tests have differences between the modes, and to
|
differences.
|
||||||
visually inspect those diagnostic differences.
|
|
||||||
|
|
||||||
If in the rare case you encounter a test that has different behavior, you can
|
If in the rare case you encounter a test that has different behavior, you can
|
||||||
run something like the following to generate the alternate stderr file:
|
run something like the following to generate the alternate stderr file:
|
||||||
|
|
@ -533,13 +523,15 @@ Currently none of the compare modes are checked in CI for UI tests.
|
||||||
|
|
||||||
## `rustc_*` TEST attributes
|
## `rustc_*` TEST attributes
|
||||||
|
|
||||||
The compiler defines several perma-unstable `#[rustc_*]` attributes gated behind the internal feature
|
The compiler defines several perma-unstable `#[rustc_*]` attributes gated behind
|
||||||
`rustc_attrs` that dump extra compiler-internal information. See the corresponding subsection in
|
the internal feature `rustc_attrs` that dump extra compiler-internal
|
||||||
[compiler debugging] for more details.
|
information. See the corresponding subsection in [compiler debugging] for more
|
||||||
|
details.
|
||||||
|
|
||||||
They can be used in tests to more precisely, legibly and easily test internal compiler state in cases
|
They can be used in tests to more precisely, legibly and easily test internal
|
||||||
where it would otherwise be very hard to do the same with "user-facing" Rust alone. Indeed, one could
|
compiler state in cases where it would otherwise be very hard to do the same
|
||||||
say that this slightly abuses the term "UI" (*user* interface) and turns such UI tests from black-box
|
with "user-facing" Rust alone. Indeed, one could say that this slightly abuses
|
||||||
tests into white-box ones. Use them carefully and sparingly.
|
the term "UI" (*user* interface) and turns such UI tests from black-box tests
|
||||||
|
into white-box ones. Use them carefully and sparingly.
|
||||||
|
|
||||||
[compiler debugging]: ../compiler-debugging.md#rustc_test-attributes
|
[compiler debugging]: ../compiler-debugging.md#rustc_test-attributes
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue