Merge from rustc

src/SUMMARY.md

@@ -75,6 +75,7 @@
 - [Prologue](./building/bootstrapping/intro.md)
 - [What Bootstrapping does](./building/bootstrapping/what-bootstrapping-does.md)
 - [How Bootstrap does it](./building/bootstrapping/how-bootstrap-does-it.md)
+- [Debugging bootstrap](./building/bootstrapping/debugging-bootstrap.md)
 
 # High-level Compiler Architecture
 

src/backend/debugging.md

@@ -56,7 +56,7 @@ These tools include:
 
 By default, the Rust build system does not check for changes to the LLVM source code or
 its build configuration settings. So, if you need to rebuild the LLVM that is linked
-into `rustc`, first delete the file `llvm-finished-building`, which should be located
+into `rustc`, first delete the file `.llvm-stamp`, which should be located
 in `build/<host-triple>/llvm/`.
 
 The default rustc compilation pipeline has multiple codegen units, which is

src/building/bootstrapping/debugging-bootstrap.md

@@ -0,0 +1,100 @@
+# Debugging bootstrap
+
+> FIXME: this page could be expanded 
+
+## `tracing` in bootstrap
+
+Bootstrap has conditional [`tracing`][tracing] setup to provide structured logging.
+
+[tracing]: https://docs.rs/tracing/0.1.41/tracing/index.html
+
+### Enabling `tracing` output
+
+Bootstrap will conditionally enable `tracing` output if the `BOOTSTRAP_TRACING` env var is set.
+
+Example usage:
+
+```bash
+$ BOOTSTRAP_TRACING=TRACE ./x build library --stage 1
+```
+
+Example output[^experimental]:
+
+![Example bootstrap tracing output](./debugging-bootstrap/tracing-output-example.png)
+
+[^experimental]: This shows what's *possible* with the infra in an experimental implementation.
+
+The env var `BOOTSTRAP_TRACING` accepts a [`tracing` env-filter][tracing-env-filter]. The `TRACE` filter will enable *all* `trace` level or less verbose level tracing output.
+
+[tracing-env-filter]: https://docs.rs/tracing-subscriber/0.3.19/tracing_subscriber/filter/struct.EnvFilter.html
+
+### Using `tracing` in bootstrap
+
+Both `tracing::*` macros and the `tracing::instrument` proc-macro attribute need to be gated behind `tracing` feature. Examples:
+
+```rs
+#[cfg(feature = "tracing")]
+use tracing::{instrument, trace};
+
+struct Foo;
+
+impl Step for Foo {
+    type Output = ();
+
+    #[cfg_attr(feature = "tracing", instrument(level = "trace", name = "Foo::should_run", skip_all))]
+    fn should_run(run: ShouldRun<'_>) -> ShouldRun<'_> {
+        #[cfg(feature = "tracing")]
+        trace!(?run, "entered Foo::should_run");
+
+        todo!()
+    }
+
+    #[cfg_attr(
+        feature = "tracing",
+        instrument(
+            level = "trace",
+            name = "Foo::run",
+            skip_all,
+            fields(compiler = ?builder.compiler),
+        ),
+    )]
+    fn run(self, builder: &Builder<'_>) -> Self::Output {
+        #[cfg(feature = "tracing")]
+        trace!(?run, "entered Foo::run");
+
+        todo!()
+    }    
+}
+```
+
+For `#[instrument]`, it's recommended to:
+
+- Gate it behind `trace` level for fine-granularity, possibly `debug` level for core functions.
+- Explicitly pick an instrumentation name via `name = ".."` to distinguish between e.g. `run` of different steps.
+- Take care to not cause diverging behavior via tracing, e.g. building extra things only when tracing infra is enabled.
+
+### Enabling `tracing` bootstrap feature in rust-analyzer
+
+You can adjust your `settings.json`'s `rust-analyzer.check.overrideCommand` and `rust-analyzer.cargo.buildScripts.overrideCommand` if you want to also enable `logging` cargo feature by default in your editor. This is mostly useful if you want proper r-a completions and such when working on bootstrap itself.
+
+```json
+"rust-analyzer.check.overrideCommand": [
+    "BOOTSTRAP_TRACING=1", // <- BOOTSTRAP_TRACING=1 won't enable tracing filter, but it will activate bootstrap's `tracing` feature
+    "python3",
+    "x.py",
+    "check",
+    "--json-output",
+    "--build-dir=build-rust-analyzer"
+],
+```
+
+```json
+"rust-analyzer.cargo.buildScripts.overrideCommand": [
+    "BOOTSTRAP_TRACING=1", // <- note this
+    "python3",
+    "x.py",
+    "check",
+    "--json-output",
+    "--build-dir=build-rust-analyzer"
+],
+```

src/building/bootstrapping/intro.md

@@ -17,5 +17,8 @@ In this section, we give a high-level overview of
 [what Bootstrap does](./what-bootstrapping-does.md), followed by a high-level
 introduction to [how Bootstrap does it](./how-bootstrap-does-it.md).
 
+Additionally, see [debugging bootstrap](./debugging-bootstrap.md) to learn
+about debugging methods.
+
 [boot]: https://en.wikipedia.org/wiki/Bootstrapping_(compilers)
 [ocaml-compiler]: https://github.com/rust-lang/rust/tree/ef75860a0a72f79f97216f8aaa5b388d98da6480/src/boot

src/building/optimized-build.md

@@ -126,4 +126,4 @@ Here is an example of how can `opt-dist` be used locally (outside of CI):
 [`Environment`]: https://github.com/rust-lang/rust/blob/ee451f8faccf3050c76cdcd82543c917b40c7962/src/tools/opt-dist/src/environment.rs#L5
 
 > Note: if you want to run the actual CI pipeline, instead of running `opt-dist` locally,
-> you can execute `DEPLOY=1 src/ci/docker/run.sh dist-x86_64-linux`.
+> you can execute `python3 src/ci/github-actions/ci.py run-local dist-x86_64-linux`.

src/tests/ci.md

@@ -299,8 +299,7 @@ platform’s custom [Docker container]. This has a lot of advantages for us:
 - We can avoid reinstalling tools (like QEMU or the Android emulator) every time
   thanks to Docker image caching.
 - Users can run the same tests in the same environment locally by just running
-  `src/ci/docker/run.sh image-name`, which is awesome to debug failures. Note
+  `python3 src/ci/github-actions/ci.py run-local <job-name>`, which is awesome to debug failures. Note that there are only linux docker images available locally due to licensing and
-  that there are only linux docker images available locally due to licensing and
   other restrictions.
 
 The docker images prefixed with `dist-` are used for building artifacts while

src/tests/directives.md

@@ -94,7 +94,8 @@ for more details.
 | Directive                         | Explanation                                                                                                              | Supported test suites                        | Possible values                                                                         |
 |-----------------------------------|--------------------------------------------------------------------------------------------------------------------------|----------------------------------------------|-----------------------------------------------------------------------------------------|
 | `check-run-results`               | Check run test binary `run-{pass,fail}` output snapshot                                                                  | `ui`, `crashes`, `incremental` if `run-pass` | N/A                                                                                     |
-| `error-pattern`                   | Check that output contains a regex pattern                                                                               | `ui`, `crashes`, `incremental` if `run-pass` | Regex                                                                                   |
+| `error-pattern`                   | Check that output contains a specific string                                                                             | `ui`, `crashes`, `incremental` if `run-pass` | String                                                                                   |
+| `regex-error-pattern`             | Check that output contains a regex pattern                                                                               | `ui`, `crashes`, `incremental` if `run-pass` | Regex                                                                                   |
 | `check-stdout`                    | Check `stdout` against `error-pattern`s from running test binary[^check_stdout]                                          | `ui`, `crashes`, `incremental`               | N/A                                                                                     |
 | `normalize-stderr-32bit`          | Normalize actual stderr (for 32-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`                          | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |
 | `normalize-stderr-64bit`          | Normalize actual stderr (for 64-bit platforms) with a rule `"<raw>" -> "<normalized>"` before comparing against snapshot | `ui`, `incremental`                          | `"<RAW>" -> "<NORMALIZED>"`, `<RAW>`/`<NORMALIZED>` is regex capture and replace syntax |

src/tests/docker.md

@@ -45,6 +45,15 @@ Some additional notes about using the Docker images:
   containers. With the container name, run `docker exec -it <CONTAINER>
   /bin/bash` where `<CONTAINER>` is the container name like `4ba195e95cef`.
 
+The approach described above is a relatively low-level interface for running the Docker images
+directly. If you want to run a full CI Linux job locally with Docker, in a way that is as close to CI as possible, you can use the following command:
+
+```bash
+python3 src/ci/github-actions/ci.py run-local <job-name>
+# For example:
+python3 src/ci/github-actions/ci.py run-local dist-x86_64-linux-alt
+```
+
 [Docker]: https://www.docker.com/
 [`src/ci/docker`]: https://github.com/rust-lang/rust/tree/master/src/ci/docker
 [`src/ci/docker/run.sh`]: https://github.com/rust-lang/rust/blob/master/src/ci/docker/run.sh