Edit "What the compiler does to your code" (#1306)
* Edit overview.md * Fix lexer crate * Edit wording Co-authored-by: pierwill <pierwill@users.noreply.github.com>
This commit is contained in:
pierwill
GitHub
parent
a1340e010c
commit
105bc3d35d
214
src/overview.md
214
src/overview.md
|
|
@ -17,94 +17,122 @@ So first, let's look at what the compiler does to your code. For now, we will
|
|||
avoid mentioning how the compiler implements these steps except as needed;
|
||||
we'll talk about that later.
|
||||
|
||||
- The compile process begins when a user writes a Rust source program in text
|
||||
and invokes the `rustc` compiler on it. The work that the compiler needs to
|
||||
perform is defined by command-line options. For example, it is possible to
|
||||
enable nightly features (`-Z` flags), perform `check`-only builds, or emit
|
||||
LLVM-IR rather than executable machine code. The `rustc` executable call may
|
||||
be indirect through the use of `cargo`.
|
||||
- Command line argument parsing occurs in the [`rustc_driver`]. This crate
|
||||
defines the compile configuration that is requested by the user and passes it
|
||||
to the rest of the compilation process as a [`rustc_interface::Config`].
|
||||
- The raw Rust source text is analyzed by a low-level lexer located in
|
||||
[`rustc_lexer`]. At this stage, the source text is turned into a stream of
|
||||
atomic source code units known as _tokens_. The lexer supports the
|
||||
Unicode character encoding.
|
||||
- The token stream passes through a higher-level lexer located in
|
||||
[`rustc_parse`] to prepare for the next stage of the compile process. The
|
||||
[`StringReader`] struct is used at this stage to perform a set of validations
|
||||
and turn strings into interned symbols (_interning_ is discussed later).
|
||||
[String interning] is a way of storing only one immutable
|
||||
copy of each distinct string value.
|
||||
### Invocation
|
||||
|
||||
- The lexer has a small interface and doesn't depend directly on the
|
||||
diagnostic infrastructure in `rustc`. Instead it provides diagnostics as plain
|
||||
data which are emitted in `rustc_parse::lexer::mod` as real diagnostics.
|
||||
- The lexer preserves full fidelity information for both IDEs and proc macros.
|
||||
- The parser [translates the token stream from the lexer into an Abstract Syntax
|
||||
Tree (AST)][parser]. It uses a recursive descent (top-down) approach to syntax
|
||||
analysis. The crate entry points for the parser are the
|
||||
[`Parser::parse_crate_mod()`][parse_crate_mod] and [`Parser::parse_mod()`][parse_mod]
|
||||
methods found in [`rustc_parse::parser::Parser`]. The external module parsing
|
||||
entry point is [`rustc_expand::module::parse_external_mod`][parse_external_mod].
|
||||
And the macro parser entry point is [`Parser::parse_nonterminal()`][parse_nonterminal].
|
||||
- Parsing is performed with a set of `Parser` utility methods including `fn bump`,
|
||||
`fn check`, `fn eat`, `fn expect`, `fn look_ahead`.
|
||||
- Parsing is organized by the semantic construct that is being parsed. Separate
|
||||
`parse_*` methods can be found in [`rustc_parse` `parser`][rustc_parse_parser_dir]
|
||||
directory. The source file name follows the construct name. For example, the
|
||||
following files are found in the parser:
|
||||
- `expr.rs`
|
||||
- `pat.rs`
|
||||
- `ty.rs`
|
||||
- `stmt.rs`
|
||||
- This naming scheme is used across many compiler stages. You will find
|
||||
either a file or directory with the same name across the parsing, lowering,
|
||||
type checking, THIR lowering, and MIR building sources.
|
||||
- Macro expansion, AST validation, name resolution, and early linting takes place
|
||||
during this stage of the compile process.
|
||||
- The parser uses the standard `DiagnosticBuilder` API for error handling, but we
|
||||
try to recover, parsing a superset of Rust's grammar, while also emitting an error.
|
||||
- `rustc_ast::ast::{Crate, Mod, Expr, Pat, ...}` AST nodes are returned from the parser.
|
||||
- We then take the AST and [convert it to High-Level Intermediate
|
||||
Representation (HIR)][hir]. This is a compiler-friendly representation of the
|
||||
AST. This involves a lot of desugaring of things like loops and `async fn`.
|
||||
- We use the HIR to do [type inference] (the process of automatic
|
||||
detection of the type of an expression), [trait solving] (the process
|
||||
of pairing up an impl with each reference to a trait), and [type
|
||||
checking] (the process of converting the types found in the HIR
|
||||
(`hir::Ty`), which represent the syntactic things that the user wrote,
|
||||
into the internal representation used by the compiler (`Ty<'tcx>`),
|
||||
and using that information to verify the type safety, correctness and
|
||||
coherence of the types used in the program).
|
||||
- The HIR is then [lowered to Mid-Level Intermediate Representation (MIR)][mir].
|
||||
- Along the way, we construct the THIR, which is an even more desugared HIR.
|
||||
THIR is used for pattern and exhaustiveness checking. It is also more
|
||||
convenient to convert into MIR than HIR is.
|
||||
- The MIR is used for [borrow checking].
|
||||
- We (want to) do [many optimizations on the MIR][mir-opt] because it is still
|
||||
generic and that improves the code we generate later, improving compilation
|
||||
speed too.
|
||||
- MIR is a higher level (and generic) representation, so it is easier to do
|
||||
some optimizations at MIR level than at LLVM-IR level. For example LLVM
|
||||
doesn't seem to be able to optimize the pattern the [`simplify_try`] mir
|
||||
opt looks for.
|
||||
- Rust code is _monomorphized_, which means making copies of all the generic
|
||||
code with the type parameters replaced by concrete types. To do
|
||||
this, we need to collect a list of what concrete types to generate code for.
|
||||
This is called _monomorphization collection_.
|
||||
- We then begin what is vaguely called _code generation_ or _codegen_.
|
||||
- The [code generation stage (codegen)][codegen] is when higher level
|
||||
representations of source are turned into an executable binary. `rustc`
|
||||
uses LLVM for code generation. The first step is to convert the MIR
|
||||
to LLVM Intermediate Representation (LLVM IR). This is where the MIR
|
||||
is actually monomorphized, according to the list we created in the
|
||||
previous step.
|
||||
- The LLVM IR is passed to LLVM, which does a lot more optimizations on it.
|
||||
It then emits machine code. It is basically assembly code with additional
|
||||
low-level types and annotations added. (e.g. an ELF object or wasm).
|
||||
- The different libraries/binaries are linked together to produce the final
|
||||
binary.
|
||||
Compilation begins when a user writes a Rust source program in text
|
||||
and invokes the `rustc` compiler on it. The work that the compiler needs to
|
||||
perform is defined by command-line options. For example, it is possible to
|
||||
enable nightly features (`-Z` flags), perform `check`-only builds, or emit
|
||||
LLVM-IR rather than executable machine code. The `rustc` executable call may
|
||||
be indirect through the use of `cargo`.
|
||||
|
||||
Command line argument parsing occurs in the [`rustc_driver`]. This crate
|
||||
defines the compile configuration that is requested by the user and passes it
|
||||
to the rest of the compilation process as a [`rustc_interface::Config`].
|
||||
|
||||
### Lexing and parsing
|
||||
|
||||
The raw Rust source text is analyzed by a low-level *lexer* located in
|
||||
[`rustc_lexer`]. At this stage, the source text is turned into a stream of
|
||||
atomic source code units known as _tokens_. The lexer supports the
|
||||
Unicode character encoding.
|
||||
|
||||
The token stream passes through a higher-level lexer located in
|
||||
[`rustc_parse`] to prepare for the next stage of the compile process. The
|
||||
[`StringReader`] struct is used at this stage to perform a set of validations
|
||||
and turn strings into interned symbols (_interning_ is discussed later).
|
||||
[String interning] is a way of storing only one immutable
|
||||
copy of each distinct string value.
|
||||
|
||||
The lexer has a small interface and doesn't depend directly on the
|
||||
diagnostic infrastructure in `rustc`. Instead it provides diagnostics as plain
|
||||
data which are emitted in `rustc_parse::lexer` as real diagnostics.
|
||||
The lexer preserves full fidelity information for both IDEs and proc macros.
|
||||
|
||||
The *parser* [translates the token stream from the lexer into an Abstract Syntax
|
||||
Tree (AST)][parser]. It uses a recursive descent (top-down) approach to syntax
|
||||
analysis. The crate entry points for the parser are the
|
||||
[`Parser::parse_crate_mod()`][parse_crate_mod] and [`Parser::parse_mod()`][parse_mod]
|
||||
methods found in [`rustc_parse::parser::Parser`]. The external module parsing
|
||||
entry point is [`rustc_expand::module::parse_external_mod`][parse_external_mod].
|
||||
And the macro parser entry point is [`Parser::parse_nonterminal()`][parse_nonterminal].
|
||||
|
||||
Parsing is performed with a set of `Parser` utility methods including `bump`,
|
||||
`check`, `eat`, `expect`, `look_ahead`.
|
||||
|
||||
Parsing is organized by semantic construct. Separate
|
||||
`parse_*` methods can be found in the [`rustc_parse`][rustc_parse_parser_dir]
|
||||
directory. The source file name follows the construct name. For example, the
|
||||
following files are found in the parser:
|
||||
|
||||
- `expr.rs`
|
||||
- `pat.rs`
|
||||
- `ty.rs`
|
||||
- `stmt.rs`
|
||||
|
||||
This naming scheme is used across many compiler stages. You will find
|
||||
either a file or directory with the same name across the parsing, lowering,
|
||||
type checking, THIR lowering, and MIR building sources.
|
||||
|
||||
Macro expansion, AST validation, name resolution, and early linting also take place
|
||||
during this stage.
|
||||
|
||||
The parser uses the standard `DiagnosticBuilder` API for error handling, but we
|
||||
try to recover, parsing a superset of Rust's grammar, while also emitting an error.
|
||||
`rustc_ast::ast::{Crate, Mod, Expr, Pat, ...}` AST nodes are returned from the parser.
|
||||
|
||||
### HIR lowering
|
||||
|
||||
We next take the AST and convert it to [High-Level Intermediate
|
||||
Representation (HIR)][hir], a more compiler-friendly representation of the
|
||||
AST. This process called "lowering". It involves a lot of desugaring of things
|
||||
like loops and `async fn`.
|
||||
|
||||
We then use the HIR to do [*type inference*] (the process of automatic
|
||||
detection of the type of an expression), [*trait solving*] (the process
|
||||
of pairing up an impl with each reference to a trait), and [*type
|
||||
checking*]. Type checking is the process of converting the types found in the HIR
|
||||
([`hir::Ty`]), which represent what the user wrote,
|
||||
into the internal representation used by the compiler ([`Ty<'tcx>`]).
|
||||
That information is usedto verify the type safety, correctness and
|
||||
coherence of the types used in the program.
|
||||
|
||||
### MIR lowering
|
||||
|
||||
The HIR is then [lowered to Mid-level Intermediate Representation (MIR)][mir],
|
||||
which is used for [borrow checking].
|
||||
|
||||
Along the way, we also construct the THIR, which is an even more desugared HIR.
|
||||
THIR is used for pattern and exhaustiveness checking. It is also more
|
||||
convenient to convert into MIR than HIR is.
|
||||
|
||||
We do [many optimizations on the MIR][mir-opt] because it is still
|
||||
generic and that improves the code we generate later, improving compilation
|
||||
speed too.
|
||||
MIR is a higher level (and generic) representation, so it is easier to do
|
||||
some optimizations at MIR level than at LLVM-IR level. For example LLVM
|
||||
doesn't seem to be able to optimize the pattern the [`simplify_try`] mir
|
||||
opt looks for.
|
||||
|
||||
Rust code is _monomorphized_, which means making copies of all the generic
|
||||
code with the type parameters replaced by concrete types. To do
|
||||
this, we need to collect a list of what concrete types to generate code for.
|
||||
This is called _monomorphization collection_ and it happens at the MIR level.
|
||||
|
||||
### Code generation
|
||||
|
||||
We then begin what is vaguely called _code generation_ or _codegen_.
|
||||
The [code generation stage][codegen] is when higher level
|
||||
representations of source are turned into an executable binary. `rustc`
|
||||
uses LLVM for code generation. The first step is to convert the MIR
|
||||
to LLVM Intermediate Representation (LLVM IR). This is where the MIR
|
||||
is actually monomorphized, according to the list we created in the
|
||||
previous step.
|
||||
The LLVM IR is passed to LLVM, which does a lot more optimizations on it.
|
||||
It then emits machine code. It is basically assembly code with additional
|
||||
low-level types and annotations added (e.g. an ELF object or WASM).
|
||||
The different libraries/binaries are then linked together to produce the final
|
||||
binary.
|
||||
|
||||
[String interning]: https://en.wikipedia.org/wiki/String_interning
|
||||
[`rustc_lexer`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_lexer/index.html
|
||||
|
|
@ -115,9 +143,9 @@ we'll talk about that later.
|
|||
[`rustc_parse`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/index.html
|
||||
[parser]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/index.html
|
||||
[hir]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/index.html
|
||||
[type inference]: https://rustc-dev-guide.rust-lang.org/type-inference.html
|
||||
[trait solving]: https://rustc-dev-guide.rust-lang.org/traits/resolution.html
|
||||
[type checking]: https://rustc-dev-guide.rust-lang.org/type-checking.html
|
||||
[*type inference*]: https://rustc-dev-guide.rust-lang.org/type-inference.html
|
||||
[*trait solving*]: https://rustc-dev-guide.rust-lang.org/traits/resolution.html
|
||||
[*type checking*]: https://rustc-dev-guide.rust-lang.org/type-checking.html
|
||||
[mir]: https://rustc-dev-guide.rust-lang.org/mir/index.html
|
||||
[borrow checking]: https://rustc-dev-guide.rust-lang.org/borrow_check.html
|
||||
[mir-opt]: https://rustc-dev-guide.rust-lang.org/mir/optimizations.html
|
||||
|
|
@ -129,6 +157,8 @@ we'll talk about that later.
|
|||
[`rustc_parse::parser::Parser`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_parse/parser/struct.Parser.html
|
||||
[parse_external_mod]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/module/fn.parse_external_mod.html
|
||||
[rustc_parse_parser_dir]: https://github.com/rust-lang/rust/tree/master/compiler/rustc_parse/src/parser
|
||||
[`hir::Ty`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/hir/struct.Ty.html
|
||||
[`Ty<'tcx>`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/ty/struct.Ty.html
|
||||
|
||||
## How it does it
|
||||
|
||||
|
|
@ -323,6 +353,7 @@ For more details on bootstrapping, see
|
|||
[_bootstrapping_]: https://en.wikipedia.org/wiki/Bootstrapping_(compilers)
|
||||
[rustc-bootstrap]: building/bootstrapping.md
|
||||
|
||||
<!--
|
||||
# Unresolved Questions
|
||||
|
||||
- Does LLVM ever do optimizations in debug builds?
|
||||
|
|
@ -332,7 +363,8 @@ For more details on bootstrapping, see
|
|||
- What is the main source entry point for `X`?
|
||||
- Where do phases diverge for cross-compilation to machine code across
|
||||
different platforms?
|
||||
|
||||
-->
|
||||
|
||||
# References
|
||||
|
||||
- Command line parsing
|
||||
|
|
|
|||
Loading…
Reference in New Issue