stabilize stage management for rustc tools
https://github.com/rust-lang/rust/pull/135990 got out of control due to excessive complexity. This PR aims to achieve the same goal with a simpler approach, likely through multiple smaller PRs. I will keep the other one read-only and open as a reference for future work.
This work stabilizes the staging logic for `ToolRustc` programs, so you no longer need to handle build and target compilers separately in steps. Previously, most tools didn't do this correctly, which was causing the compiler to be built twice (e.g., `x test cargo --stage 1` would compile the stage 2 compiler before, but now it only compiles the stage 1 compiler).
I also tried to document how we should write `ToolRustc` steps as they are quite different and require more attention than other tools.
Next goal is to stabilize how stages are handled for the rustc itself. Currently, `x build --stage 1` builds the stage 1 compiler which is fine, but `x build compiler --stage 1` builds stage 2 compiler.
~~for now, r? ghost~~
Added project-specific Zed IDE settings
This repository currently has project-specific VS Code IDE settings in `.vscode` and `compiler/rustc_codegen_cranelift/.vscode`. Now there are equivalent project-specific Zed IDE settings alongside those.
This fixes `rust-analyzer` not being able to properly handle this project.
Note that:
1. The contents of `src/tools/rust-analyzer/.vscode` could not be translated to Zed, as they aren't basic IDE settings.
2. One of the VS Code settings in `.vscode` has no corresponding setting in Zed, and so this has been noted like this:
```json
"_settings_only_in_vs_code_not_yet_in_zed": {
"git.detectSubmodulesLimit": 20
},
```
While there were comments indicating which nightly versions the examples
were tested with, those versions did not work for me: neither did the
examples compile, nor did they produce the expected output.
This commit fixes the compilation issues, using nightly-2025-02-13 for
all examples (previously the version differed between the examples) and,
in the case of the `rustc_driver` examples, also fixes the argument
passing: rustc ignores the first argument, so we need to pass the
filename as the second (otherwise we only get the help text printed).
Note that the `rustc-interface-getting-diagnostics.rs` example still
does not produce any output, which I assume is not how it is intended.
However, I don't know enough to fix it.
To avoid inconsistencies between the documented version and the actually
required version I've moved the version comment from the Markdown into
the Rust code where it hopefully won't be forgotten as easily.
Finally I've clarified in the examples' README that you also need to use
the proper nightly version when compiling the examples, not just when
running them.
Add profiling of bootstrap commands using Chrome events
Since we now have support for tracing in bootstrap, and the execution of most commands is centralized within a few functions, it's quite trivial to also trace command execution, and visualize it using the Chrome profiler. This can be helpful both to profile what takes time in bootstrap and also to get a visual idea of what happens in a given bootstrap invocation (since the execution of external commands is usually the most interesting thing).
This is how it looks:
I first tried to use [tracing-flame](https://github.com/tokio-rs/tracing/tree/master/tracing-flame), but the output wasn't very useful, because the event/stackframe names were bootstrap code locations, instead of the command contents.
r? ``@jieyouxu``
valtree performance tuning
Summary: This PR makes type checking of code with many type-level constants faster.
After https://github.com/rust-lang/rust/pull/136180 was merged, we observed a small perf regression (https://github.com/rust-lang/rust/pull/136318#issuecomment-2635562821). This happened because that PR introduced additional copies in the fast reject code path for consts, which is very hot for certain crates: 6c1d960d88/compiler/rustc_type_ir/src/fast_reject.rs (L486-L487)
This PR improves the performance again by properly interning the valtrees so that copying and comparing them becomes faster. This will become especially useful with `feature(adt_const_params)`, so the fast reject code doesn't have to do a deep compare of the valtrees.
Note that we can't just compare the interned consts themselves in the fast reject, because sometimes `'static` lifetimes in the type are be replaced with inference variables (due to canonicalization) on one side but not the other.
A less invasive alternative that I considered is simply avoiding copies introduced by https://github.com/rust-lang/rust/pull/136180 and comparing the valtrees it in-place (see commit: 9e91e50ac5 / perf results: https://github.com/rust-lang/rust/pull/136593#issuecomment-2642303245), however that was still measurably slower than interning.
There are some minor regressions in secondary benchmarks: These happen due to changes in memory allocations and seem acceptable to me. The crates that make heavy use of valtrees show no significant changes in memory usage.
Parallel-compiler-related cleanup
Parallel-compiler-related cleanup
I carefully split changes into commits. Commit messages are self-explanatory. Squashing is not recommended.
cc "Parallel Rustc Front-end" https://github.com/rust-lang/rust/issues/113349
r? SparrowLii
``@rustbot`` label: +WG-compiler-parallel
dev-guide: Link to `t-lang` procedures for new features
I was confused in https://github.com/rust-lang/rust/pull/136867, because while I did remember that such a procedure existed, but I couldn't seem to find it in the dev guide.
Improve documentation when adding a new target
https://github.com/rust-lang/rust/pull/133631#issuecomment-2607877936 shows that it can be a bit difficult process-wise to add a new target.
I've added a bit of text to the docs, suggesting that users add the target defintion/spec first, and later work on `std` support.
I also found that we have two places where we document how to add a new target. I've linked these for now, but they should probably be merged somehow in the future.
`@rustbot` label A-docs
r? compiler
CC `@workingjubilee` who's worked a lot on target specs IIRC.
Simplify and consolidate the way we handle construct `OutlivesEnvironment` for lexical region resolution
This is best reviewed commit-by-commit. I tried to consolidate the API for lexical region resolution *first*, then change the API when it was finally behind a single surface.
r? lcnr or reassign
Tidy Python improvements
Fixes display of Python formatting diffs in tidy, and refactors the code to make it simpler and more robust. Also documents Python formatting and linting in the Rustc dev guide.
Fixes: https://github.com/rust-lang/rust/issues/135942
r? `@onur-ozkan`
Implement `needs-subprocess` directive, and cleanup a bunch of tests to use `needs-{subprocess,threads}`
### Summary
Closes#128295.
- Implements `//@ needs-subprocess` directive in compiletest as requested in #128295. However, compiletest is a host tool, so we can't just try to spawn process because that spawns the process on *host*, not the *target*, under cross-compilation scenarios.
- The short-term solution is to add *Yet Another* list of allow-list targets.
- The long-term solution is to first check if a `$target` supports std, then try to run a binary to do run-time capability detection *on the target*. But that is tricky because you have to build-and-run a binary *for the target*.
- This PR picks the short-term solution, because the long-term solution is highly non-trivial, and it's already an improvement over individual `ignore-*`s all over the place.
- Opened an issue about the long-term solution in #135928.
- Documents `//@ needs-subprocess` in rustc-dev-guide.
- Replace `ignore-{wasm,wasm32,emscripten,sgx}` with `needs-{subprocess,threads}` where suitable in tests.
- Some drive-by test changes as I was trying to figure out if I could use `needs-{subprocess,threads}` and found some bits needlessly distracting.
Count of tests that use `ignore-{wasm,wasm32,emscripten,sgx}` before and after this PR:
| State | `ignore-sgx` | `ignore-wasm` | `ignore-emscripten` |
| - | - | - | - |
| Before this PR | 96 | 88 | 207 |
| After this PR | 36 | 38 | 61 |
<details>
<summary>Commands used to find out locally</summary>
```
--- before
[17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-sgx" tests | wc -l
96
[17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-wasm" tests | wc -l
88
[17:40] Joe:rust (fresh) | rg --no-ignore -l "ignore-emscripten" tests | wc -l
207
--- after
[17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-sgx" tests | wc -l
36
[17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-wasm" tests | wc -l
38
[17:39] Joe:rust (needs-subprocess-thread) | rg --no-ignore -l "ignore-emscripten" tests | wc -l
61
```
</details>
### Review advice
- Best reviewed commit-by-commit.
- Non-trivial test changes (not mechanically simple replacements) are split into individual commits to help with review. Their individual commit messages give some basic description of the changes.
- I *could* split some test changes out into another PR, but I found that I needed to change some tests to `needs-threads`, some to `needs-subprocess`, and some needed to use *both*, so they might conflict and become very annoying.
---
r? ``@ghost`` (need to run try jobs)
try-job: x86_64-msvc-1
try-job: i686-msvc-1
try-job: i686-mingw
try-job: x86_64-mingw-1
try-job: x86_64-apple-1
try-job: aarch64-apple
try-job: aarch64-gnu
try-job: test-various
try-job: armhf-gnu
remove pointless allowed_through_unstable_modules on TryFromSliceError
This got added in https://github.com/rust-lang/rust/pull/132482 but the PR does not explain why. `@lukas-code` do you still remember? Also Cc `@Noratrieb` as reviewer of that PR.
If I understand the issue description correctly, all paths under which this type is exported are stable now: `core::array::TryFromSliceError` and `std::array::TryFromSliceError`. If that is the case, we shouldn't have the attribute; it's a terrible hack that should only be used when needed to maintain backward compatibility. Getting some historic information right is IMO *not* sufficient justification to risk accidentally exposing this type via more unstable paths today or in the future.
Get rid of RunCompiler
The various `set_*` methods that have been removed can be replaced by setting the respective fields in the `Callbacks::config` implementation. `set_using_internal_features` was often forgotten and it's equivalent is now done automatically.
This documents how to determine which settings are used in CI, since I
see this question come up regularly. We currently don't have a great way
to answer the question, but at least there is something.
* Fix broken link in syntax-intro.md
* Fix broken link in tests/compiletest.md
* Fix incorrectly formatted reference-style link in compiler-src.md
* Fix broken links to rustc-driver.md to point to rustc-driver/intro.md
* Define URLs for incomplete links to rustc sources in stabilization_guide.md
* Define link to coherence.md for incomplete link in solve/invariants.md
Describe procedure for adding new crashes tests and removing crashes
tests when the crash is intentionally fixed.
Also mention setting `COMPILETEST_VERBOSE_CRASHES=1` env var for verbose
crashes test output.
bors
Deadbeef
Matthias Krüger
onur-ozkan
Ed Page
Chai T. Rex
Jakub Beránek
许杰友 Jieyou Xu (Joe)
Florian Brucker
Martin Liska
yukang
Jubilee
Jacob Pratt
jyn
Lukas Markeffsky
Guillaume Gomez
Tshepang Mbambo
Mads Marquart
Askar Safin
The rustc-dev-guide Cronjob Bot
Urgau
Michael Howell
MarcoIeni
Rehmatpal Singh
Yuki Okushi
nora
Ryan Cumming
Michael Goulet
chiichen
Boxy
yegeunyang
Joren-vanGoethem
Eric Huss
León Orell Valerian Liehr
Ada Alakbarova
bjorn3
Patrick Oliveira
lcnr
Ryan Mehri
Noah Lev
Ralf Jung
Samson
Max Heller
Stuart Cook
clubby789
Chris Bloodsworth
Connor Tsui
Orion Gonzalez
Keith Wansbrough
Camille Gillot
Taylor Cramer
Yutaro Ohno
Sam Estep
Tbkhi
Barrett
Nicholas Nethercote
DianQK
David Wood
Thomas Nyman
lolbinarycat
Juniper Tyree
Kajetan Puchalski
Yoh Deadfall
Kajetan Puchalski
Christopher Speir
Yuri Astrakhan
Tbkhi
Tbkhi
Felix Rath
dianne
trevyn
Vetle Rasmussen
joculatrix
adwin
Santiago Pastorino