Update compiler-src.md
Various link additions and minor edits for clarity. revisions fix line length adding links minor edits
This commit is contained in:
Tbkhi
nora
parent
afdad072e3
commit
b20c899ee1
|
|
@ -2,44 +2,64 @@
|
|||
|
||||
<!-- toc -->
|
||||
|
||||
Now that we have [seen what the compiler does](./overview.md), let's take a
|
||||
look at the structure of the [`rust-lang/rust`] repository, where the rustc
|
||||
source code lives.
|
||||
Now that we have [seen what the compiler does][orgch],
|
||||
let's take a look at the structure of the [`rust-lang/rust`] repository,
|
||||
where the rustc source code lives.
|
||||
|
||||
[`rust-lang/rust`]: https://github.com/rust-lang/rust
|
||||
|
||||
> You may find it helpful to read the ["Overview of the compiler"](./overview.md)
|
||||
> You may find it helpful to read the ["Overview of the compiler"][orgch]
|
||||
> chapter, which introduces how the compiler works, before this one.
|
||||
|
||||
[orgch]: ./overview.md
|
||||
|
||||
## Workspace structure
|
||||
|
||||
The `rust-lang/rust` repository consists of a single large cargo workspace
|
||||
containing the compiler, the standard libraries (`core`, `alloc`, `std`,
|
||||
`proc_macro`, etc), and `rustdoc`, along with the build system and a bunch of
|
||||
tools and submodules for building a full Rust distribution.
|
||||
The [`rust-lang/rust`] repository consists of a single large cargo workspace
|
||||
containing the compiler, the standard libraries ([`core`], [`alloc`],[ `std`],
|
||||
[`proc_macro`], [`etc`]), and [`rustdoc`], along with the build system and a
|
||||
bunch of tools and submodules for building a full Rust distribution.
|
||||
|
||||
The repository consists of three main directories:
|
||||
|
||||
- `compiler/` contains the source code for `rustc`. It consists of many crates
|
||||
- [`compiler/`] contains the source code for `rustc`. It consists of many crates
|
||||
that together make up the compiler.
|
||||
|
||||
- `library/` contains the standard libraries (`core`, `alloc`, `std`,
|
||||
`proc_macro`, `test`), as well as the Rust runtime (`backtrace`, `rtstartup`,
|
||||
`lang_start`).
|
||||
- [`library/`] contains the standard libraries ([`core`], [`alloc`], [`std`],
|
||||
[`proc_macro`], [`test`]), as well as the Rust runtime ([`backtrace`], [`rtstartup`],
|
||||
[`lang_start`]).
|
||||
|
||||
- `tests/` contains the compiler tests.
|
||||
- [`tests/`] contains the compiler tests.
|
||||
|
||||
- `src/` contains the source code for rustdoc, clippy, cargo, the build system,
|
||||
- [`src/`] contains the source code for [`rustdoc`], [`clippy`], [`cargo`], the build system,
|
||||
language docs, etc.
|
||||
|
||||
[`alloc`]: https://github.com/rust-lang/rust/tree/master/library/alloc
|
||||
[`backtrace`]: https://github.com/rust-lang/backtrace-rs/
|
||||
[`cargo`]: https://github.com/rust-lang/cargo
|
||||
[`clippy`]: https://github.com/rust-lang/rust/tree/master/src/tools/clippy
|
||||
[`compiler/`]: https://github.com/rust-lang/rust/tree/master/compiler
|
||||
[`core`]: https://github.com/rust-lang/rust/tree/master/library/core
|
||||
[`etc`]: https://github.com/rust-lang/rust/tree/master/src/etc
|
||||
[`lang_start`]: https://github.com/rust-lang/rust/blob/master/library/std/src/rt.rs
|
||||
[`library/`]: https://github.com/rust-lang/rust/tree/master/library
|
||||
[`proc_macro`]: https://github.com/rust-lang/rust/tree/master/library/proc_macro
|
||||
[`rtstartup`]: https://github.com/rust-lang/rust/tree/master/library/rtstartup
|
||||
[`rust-lang/rust`]: https://github.com/rust-lang/rust
|
||||
[`rustdoc`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc
|
||||
[`src/`]: https://github.com/rust-lang/rust/tree/master/src
|
||||
[`std`]: https://github.com/rust-lang/rust/tree/master/library/std
|
||||
[`test`]: https://github.com/rust-lang/rust/tree/master/library/test
|
||||
[`tests/`]: https://github.com/rust-lang/rust/tree/master/tests
|
||||
|
||||
## Compiler
|
||||
|
||||
The compiler is implemented in the various `compiler/` crates.
|
||||
The `compiler/` crates all have names starting with `rustc_*`. These are a
|
||||
The compiler is implemented in the various [`compiler/`] crates.
|
||||
The [`compiler/`] crates all have names starting with `rustc_*`. These are a
|
||||
collection of around 50 interdependent crates ranging in size from tiny to
|
||||
huge. There is also the `rustc` crate which is the actual binary (i.e. the
|
||||
`main` function); it doesn't actually do anything besides calling the
|
||||
`rustc_driver` crate, which drives the various parts of compilation in other
|
||||
[`rustc_driver`] crate, which drives the various parts of compilation in other
|
||||
crates.
|
||||
|
||||
The dependency structure of these crates is complex, but roughly it is
|
||||
|
|
@ -58,16 +78,16 @@ something like this:
|
|||
[`Span`]), or error reporting: [`rustc_data_structures`],
|
||||
[`rustc_span`], [`rustc_errors`], etc.
|
||||
|
||||
[main]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.main.html
|
||||
[`rustc_data_structures`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/index.html
|
||||
[`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/index.html
|
||||
[`rustc_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html
|
||||
[`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html
|
||||
[`rustc_middle`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/index.html
|
||||
[`rustc_data_structures`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/index.html
|
||||
[`rustc_span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/index.html
|
||||
[`Span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.Span.html
|
||||
[`rustc_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html
|
||||
[main]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.main.html
|
||||
|
||||
You can see the exact dependencies by reading the `Cargo.toml` for the various
|
||||
You can see the exact dependencies by reading the [`Cargo.toml`] for the various
|
||||
crates, just like a normal Rust crate.
|
||||
|
||||
One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM.
|
||||
|
|
@ -78,117 +98,110 @@ compiler can interface with it.
|
|||
Most of this book is about the compiler, so we won't have any further
|
||||
explanation of these crates here.
|
||||
|
||||
[`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src/
|
||||
[`compiler/rustc_llvm`]: https://github.com/rust-lang/rust/tree/master/compiler/rustc_llvm
|
||||
[`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src/
|
||||
[`Cargo.toml`]: https://github.com/rust-lang/rust/blob/master/Cargo.toml
|
||||
|
||||
### Big picture
|
||||
|
||||
The dependency structure is influenced by two main factors:
|
||||
The dependency structure of the compiler is influenced by two main factors:
|
||||
|
||||
1. Organization. The compiler is a _huge_ codebase; it would be an impossibly
|
||||
large crate. In part, the dependency structure reflects the code structure
|
||||
of the compiler.
|
||||
2. Compile time. By breaking the compiler into multiple crates, we can take
|
||||
2. Compile-time. By breaking the compiler into multiple crates, we can take
|
||||
better advantage of incremental/parallel compilation using cargo. In
|
||||
particular, we try to have as few dependencies between crates as possible so
|
||||
that we don't have to rebuild as many crates if you change one.
|
||||
|
||||
At the very bottom of the dependency tree are a handful of crates that are used
|
||||
by the whole compiler (e.g. [`rustc_span`]). The very early parts of the
|
||||
compilation process (e.g. parsing and the AST) depend on only these.
|
||||
compilation process (e.g. [parsing and the Abstract Syntax Tree (`AST`)][parser])
|
||||
depend on only these.
|
||||
|
||||
After the AST is constructed and other early analysis is done, the compiler's [query system][query]
|
||||
gets set up. The query system is set up in a clever way using function
|
||||
pointers. This allows us to break dependencies between crates, allowing more
|
||||
parallel compilation.
|
||||
The query system is defined in [`rustc_middle`], so nearly all
|
||||
subsequent parts of the compiler depend on this crate. It is a really large
|
||||
crate, leading to long compile times. Some efforts have been made to move stuff
|
||||
out of it with limited success. Another unfortunate side effect is that sometimes
|
||||
related functionality gets scattered across different crates. For example,
|
||||
linting functionality is scattered across earlier parts of the crate,
|
||||
[`rustc_lint`], [`rustc_middle`], and other places.
|
||||
After the [`AST`][parser] is constructed and other early analysis is done, the
|
||||
compiler's [query system][query] gets set up. The query system is set up in a
|
||||
clever way using function pointers. This allows us to break dependencies
|
||||
between crates, allowing more parallel compilation. The query system is defined
|
||||
in [`rustc_middle`], so nearly all subsequent parts of the compiler depend on
|
||||
this crate. It is a really large crate, leading to long compile times. Some
|
||||
efforts have been made to move stuff out of it with varying success. Another
|
||||
side-effect is that sometimes related functionality gets scattered across
|
||||
different crates. For example, linting functionality is found across earlier
|
||||
parts of the crate, [`rustc_lint`], [`rustc_middle`], and other places.
|
||||
|
||||
Ideally there would be fewer, more cohesive crates, with incremental and
|
||||
parallel compilation making sure compile times stay reasonable. However,
|
||||
incremental and parallel compilation haven't gotten good enough for that yet,
|
||||
so breaking things into separate crates has been our solution so far.
|
||||
|
||||
At the top of the dependency tree is [`rustc_driver`] and [`rustc_interface`]
|
||||
which is an unstable wrapper around the query system helping drive various
|
||||
stages of compilation. Other consumers of the compiler may use this interface
|
||||
in different ways (e.g. [`rustdoc`] or maybe eventually `rust-analyzer`). The
|
||||
[`rustc_driver`] crate first parses command line arguments and then uses
|
||||
[`rustc_interface`] to drive the compilation to completion.
|
||||
|
||||
[parser]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/index.html
|
||||
[`rustc_lint`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lint/index.html
|
||||
|
||||
Ideally there would be fewer, more
|
||||
cohesive crates, with incremental and parallel compilation making sure compile
|
||||
times stay reasonable. However, our incremental and parallel compilation haven't
|
||||
gotten good enough for that yet, so breaking things into separate crates has
|
||||
been our solution so far.
|
||||
|
||||
At the top of the dependency tree are the [`rustc_interface`] and
|
||||
[`rustc_driver`] crates. [`rustc_interface`] is an unstable wrapper around the
|
||||
query system that helps to drive the various stages of compilation. Other
|
||||
consumers of the compiler may use this interface in different ways (e.g.
|
||||
rustdoc or maybe eventually rust-analyzer). The [`rustc_driver`] crate first
|
||||
parses command line arguments and then uses [`rustc_interface`] to drive the
|
||||
compilation to completion.
|
||||
|
||||
[query]: ./query.md
|
||||
|
||||
[orgch]: ./overview.md
|
||||
|
||||
## rustdoc
|
||||
|
||||
The bulk of `rustdoc` is in [`librustdoc`]. However, the `rustdoc` binary
|
||||
The bulk of [`rustdoc`] is in [`librustdoc`]. However, the [`rustdoc`] binary
|
||||
itself is [`src/tools/rustdoc`], which does nothing except call [`rustdoc::main`].
|
||||
|
||||
There is also javascript and CSS for the rustdocs in [`src/tools/rustdoc-js`]
|
||||
There is also `JavaScript` and `CSS` for the docs in [`src/tools/rustdoc-js`]
|
||||
and [`src/tools/rustdoc-themes`].
|
||||
|
||||
You can read more about rustdoc in [this chapter][rustdocch].
|
||||
You can read more about [`rustdoc`] in [this chapter][`rustdoc`].
|
||||
|
||||
[`librustdoc`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/index.html
|
||||
[`rustdoc::main`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/fn.main.html
|
||||
[`src/tools/rustdoc`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc
|
||||
[`src/tools/rustdoc-js`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-js
|
||||
[`src/tools/rustdoc-themes`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-themes
|
||||
|
||||
[rustdocch]: ./rustdoc.md
|
||||
[`src/tools/rustdoc`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc
|
||||
[`rustdoc`]: ./rustdoc.md
|
||||
|
||||
## Tests
|
||||
|
||||
The test suite for all of the above is in [`tests/`]. You can read more
|
||||
about the test suite [in this chapter][testsch].
|
||||
|
||||
The test harness itself is in [`src/tools/compiletest`].
|
||||
|
||||
[testsch]: ./tests/intro.md
|
||||
The test harness is in [`src/tools/compiletest/`][`compiletest/`].
|
||||
|
||||
[`tests/`]: https://github.com/rust-lang/rust/tree/master/tests
|
||||
[`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
|
||||
[testsch]: ./tests/intro.md
|
||||
|
||||
## Build System
|
||||
|
||||
There are a number of tools in the repository just for building the compiler,
|
||||
standard library, rustdoc, etc, along with testing, building a full Rust
|
||||
standard library, [`rustdoc`], etc, along with testing, building a full Rust
|
||||
distribution, etc.
|
||||
|
||||
One of the primary tools is [`src/bootstrap`]. You can read more about
|
||||
One of the primary tools is [`src/bootstrap/`]. You can read more about
|
||||
bootstrapping [in this chapter][bootstch]. The process may also use other tools
|
||||
from `src/tools/`, such as [`tidy`] or [`compiletest`].
|
||||
|
||||
[`src/bootstrap`]: https://github.com/rust-lang/rust/tree/master/src/bootstrap
|
||||
[`tidy`]: https://github.com/rust-lang/rust/tree/master/src/tools/tidy
|
||||
[`compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
|
||||
from [`src/tools/`], such as [`tidy/`] or [`compiletest/`].
|
||||
|
||||
[`compiletest/`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest
|
||||
[`src/bootstrap/`]: https://github.com/rust-lang/rust/tree/master/src/bootstrap
|
||||
[`src/tools/`]: https://github.com/rust-lang/rust/tree/master/src/tools
|
||||
[`tidy/`]: https://github.com/rust-lang/rust/tree/master/src/tools/tidy
|
||||
[bootstch]: ./building/bootstrapping/intro.md
|
||||
|
||||
## Standard library
|
||||
|
||||
The standard library crates are all in `library/`. They have intuitive names
|
||||
like `std`, `core`, `alloc`, etc. There is also `proc_macro`, `test`, and
|
||||
other runtime libraries.
|
||||
|
||||
This code is fairly similar to most other Rust crates except that it must be
|
||||
built in a special way because it can use unstable features.
|
||||
built in a special way because it can use unstable ([`nightly`]) features.
|
||||
The standard library is sometimes referred to as [`libstd or the "standard facade"`].
|
||||
|
||||
[`libstd or the "standard facade"`]: https://rust-lang.github.io/rfcs/0040-libstd-facade.html
|
||||
[`nightly`]: https://doc.rust-lang.org/nightly/nightly-rustc/
|
||||
|
||||
## Other
|
||||
|
||||
There are a lot of other things in the `rust-lang/rust` repo that are related
|
||||
to building a full Rust distribution. Most of the time you don't need to worry
|
||||
about them.
|
||||
to building a full Rust distribution. Most of the time you don't need to worry about them.
|
||||
|
||||
These include:
|
||||
- [`src/ci`]: The CI configuration. This actually quite extensive because we
|
||||
|
|
|
|||
Loading…
Reference in New Issue