Reorganize the guide (#651)

* start on guide reorg

* backend intro

* part 2, 3, 4 intros

* some missing files

* Fix typos

Co-Authored-By: Yuki Okushi <huyuumi.dev@gmail.com>
Co-Authored-By: Chris Simpkins <git.simpkins@gmail.com>

* add links from intro

* rename compiler source chapter and add placeholder for overview

* rename parts, mv rustc_driver to part 3, make syntax chapter

* update part intros

* add syntax ch intro

* address review comments

* Better overview chapter placeholder

Co-Authored-By: Chris Simpkins <git.simpkins@gmail.com>

* fix link

Co-authored-by: Yuki Okushi <huyuumi.dev@gmail.com>
Co-authored-by: Chris Simpkins <git.simpkins@gmail.com>

src/SUMMARY.md

@@ -33,11 +33,10 @@
         - ["Cleanup Crew" ICE-breakers](ice-breaker/cleanup-crew.md)
         - [LLVM ICE-breakers](ice-breaker/llvm.md)
     - [Licenses](./licenses.md)
-- [Part 2: How rustc works](./part-2-intro.md)
-    - [High-level overview of the compiler source](./high-level-overview.md)
-    - [The Rustc Driver and Interface](./rustc-driver.md)
-        - [Rustdoc](./rustdoc.md)
-        - [Ex: Type checking through `rustc_interface`](./rustc-driver-interacting-with-the-ast.md)
+
+- [Part 2: High-level Compiler Architecture](./part-2-intro.md)
+    - [Overview of the Compiler](./overview.md)
+    - [The compiler source code](./compiler-src.md)
     - [Queries: demand-driven compilation](./query.md)
         - [The Query Evaluation Model in Detail](./queries/query-evaluation-model-in-detail.md)
         - [Incremental compilation](./queries/incremental-compilation.md)
@@ -46,15 +45,29 @@
         - [Profiling Queries](./queries/profiling.md)
         - [Salsa](./salsa.md)
     - [Memory Management in Rustc](./memory.md)
+
+- [Part 3: Source Code Representations](./part-3-intro.md)
+    - [The Rustc Driver and Interface](./rustc-driver.md)
+        - [Rustdoc](./rustdoc.md)
+        - [Ex: Type checking through `rustc_interface`](./rustc-driver-interacting-with-the-ast.md)
+    - [Syntax and the AST](./syntax-intro.md)
         - [Lexing and Parsing](./the-parser.md)
         - [`#[test]` Implementation](./test-implementation.md)
         - [Panic Implementation](./panic-implementation.md)
         - [Macro expansion](./macro-expansion.md)
         - [Name resolution](./name-resolution.md)
+        - [AST Validation](./ast-validation.md)
+        - [Feature Gate Checking](./feature-gate-ck.md)
     - [The HIR (High-level IR)](./hir.md)
         - [Lowering AST to HIR](./lowering.md)
         - [Debugging](./hir-debugging.md)
+    - [The MIR (Mid-level IR)](./mir/index.md)
+        - [HAIR and MIR construction](./mir/construction.md)
+        - [MIR visitor and traversal](./mir/visitor.md)
+        - [MIR passes: getting the MIR for a function](./mir/passes.md)
     - [Closure expansion](./closure.md)
+
+- [Part 4: Analysis](./part-4-intro.md)
     - [The `ty` module: representing types](./ty.md)
         - [Generics and substitutions](./generics.md)
         - [`TypeFolder` and `TypeFoldable`](./ty-fold.md)
@@ -82,12 +95,7 @@
         - [Method Lookup](./method-lookup.md)
         - [Variance](./variance.md)
         - [Opaque Types](./opaque-types-type-alias-impl-trait.md)
-    - [The MIR (Mid-level IR)](./mir/index.md)
-        - [MIR construction](./mir/construction.md)
-        - [MIR visitor and traversal](./mir/visitor.md)
-        - [MIR passes: getting the MIR for a function](./mir/passes.md)
-        - [MIR optimizations](./mir/optimizations.md)
-        - [Debugging](./mir/debugging.md)
+    - [Pattern and Exhaustiveness Checking](./pat-exhaustive-checking.md)
     - [The borrow checker](./borrow_check.md)
         - [Tracking moves and initialization](./borrow_check/moves_and_initialization.md)
             - [Move paths](./borrow_check/moves_and_initialization/move_paths.md)
@@ -100,10 +108,14 @@
             - [Closure constraints](./borrow_check/region_inference/closure_constraints.md)
             - [Error reporting](./borrow_check/region_inference/error_reporting.md)
         - [Two-phase-borrows](./borrow_check/two_phase_borrows.md)
+    - [Parameter Environments](./param_env.md)
+
+- [Part 5: From MIR to binaries](./part-5-intro.md)
+    - [The MIR (Mid-level IR)](./mir/index.md)
+        - [MIR optimizations](./mir/optimizations.md)
+        - [Debugging](./mir/debugging.md)
     - [Constant evaluation](./const-eval.md)
         - [miri const evaluator](./miri.md)
-    - [Parameter Environments](./param_env.md)
-    - [Compiler Backend](./backend/backend.md)
     - [Monomorphization](./backend/monomorph.md)
     - [Lowering MIR](./backend/lowering-mir.md)
     - [Code Generation](./backend/codegen.md)

src/about-this-guide.md

@@ -2,14 +2,29 @@
 
 This guide is meant to help document how rustc – the Rust compiler –
 works, as well as to help new contributors get involved in rustc
-development. It is not meant to replace code documentation – each
-chapter gives only high-level details, the kinds of things that
-(ideally) don't change frequently.
+development.
 
-There are three parts to this guide. Part 1 contains information that should
-be useful no matter how you are contributing. Part 2 contains information
-about how the compiler works. Finally, there are some appendices at the
-end with useful reference information.
+There are six parts to this guide:
+
+1. [Contributing][p1]: Contains information that should be useful no matter how
+   you are contributing, such as procedures for contribution, building the
+   compiler, etc.
+2. [High-level Compiler Architecture][p2]: Discusses the high-level
+   architecture of the compiler, especially the query system.
+3. [The Compiler Frontend][p3]: Discusses the compiler frontend and internal
+   representations.
+4. [The Type System][p4]: Discusses the type system.
+5. [The Compiler Backend][p5]: Discusses the compiler backend, code generation,
+   linking, and debug info.
+6. [Appendices][app] at the end with useful reference information. There are a
+   few of these with different information, inluding a glossary.
+
+[p1]: ./part-1-intro.md
+[p2]: ./part-2-intro.md
+[p3]: ./part-3-intro.md
+[p4]: ./part-4-intro.md
+[p5]: ./part-5-intro.md
+[app]: ./appendix/background.md
 
 The guide itself is of course open-source as well, and the sources can
 be found at the [GitHub repository]. If you find any mistakes in the

src/ast-validation.md

@@ -0,0 +1,6 @@
+# AST Validation
+
+AST validation is the process of checking various correctness properties about
+the AST after macro expansion.
+
+**TODO**: write this chapter.

src/feature-gate-ck.md

@@ -0,0 +1,3 @@
+# Feature Gate Checking
+
+**TODO**: this chapter

src/mir/construction.md

@@ -1,4 +1,4 @@
-# MIR construction
+# HAIR and MIR construction
 
 The lowering of [HIR] to [MIR] occurs for the following (probably incomplete)
 list of items:

src/overview.md

@@ -0,0 +1,3 @@
+# Overview of the Compiler
+
+Coming soon!  Work is in progress on this chapter.  See https://github.com/rust-lang/rustc-dev-guide/pull/633 for the source and the [project README](https://github.com/rust-lang/rustc-dev-guide) for local build instructions.

src/part-2-intro.md

@@ -1,12 +1,15 @@
 # Part 2: How rustc works
 
-This part of the guide describes how the compiler works. It goes through
-everything from high-level structure of the compiler to how each stage of
-compilation works.
+The remaining parts of this guide discuss how the compiler works. They go
+through everything from high-level structure of the compiler to how each stage
+of compilation works. They should be friendly to both readers interested in the
+end-to-end process of compilation _and_ readers interested in learning about a
+specific system they wish to contribute to. If anything is unclear, feel free
+to file an issue on the [rustc-dev-guide
+repo](https://github.com/rust-lang/rustc-dev-guide/issues) or contact the compiler
+team, as detailed in [this chapter from Part 1](./compiler-team.md).
 
-This section should be friendly to both readers interested in the end-to-end
-process of compilation _and_ readers interested in learning about a specific
-system they wish to contribute to. If anything is unclear, feel free to file
-an issue on the [rustc-dev-guide repo](https://github.com/rust-lang/rustc-dev-guide)
-or contact the compiler team, as detailed in [this chapter from Part
-1](./compiler-team.md).
+In this part, we will specifically look at the high-level architecture of the
+compiler. Specifically, will look at the query system, incremental compilation,
+and interning. These are three overarching design choices that impact the whole
+compiler.

src/part-3-intro.md

@@ -0,0 +1,8 @@
+# Part 3: Source Code Representations
+
+This part describes the process of taking raw source code from the user and
+transforming it into various forms that the compiler can work with easily.
+These are called intermediate representations.
+
+This process starts with compiler understanding what the user has asked for:
+parsing the command line arguments given and determining what it is to compile.

src/part-4-intro.md

@@ -0,0 +1,12 @@
+# Part 4: Analysis
+
+This part discusses the many analyses that the compiler uses to check various
+properties of the code and to inform later stages. Typically, this is what people
+mean when they talk about "Rust's type system". This includes the
+representation, inference, and checking of types, the trait system, and the
+borrow checker. These analyses do not happen as one big pass or set of
+contiguous passes. Rather, they are spread out throughout various parts of the
+compilation process and use different intermediate representations. For example,
+type checking happens on the HIR, while borrow checking happens on the MIR.
+Nonetheless, for the sake of presentation, we will discuss all of these
+analyses in this part of the guide.

src/part-5-intro.md

@@ -1,19 +1,20 @@
-# The Compiler Backend
+# From MIR to binaries
 
 All of the preceding chapters of this guide have one thing in common: we never
 generated any executable machine code at all! With this chapter, all of that
 changes.
 
-It's often useful to think of compilers as being composed of a _frontend_ and a
-_backend_  (though in rustc, there's not a sharp line between frontend and
-backend). The _frontend_ is responsible for taking raw source code, checking it
-for correctness, and getting it into a format usable by the backend. For rustc,
-this format is the MIR.  The _backend_ refers to the parts of the compiler that
-turn rustc's MIR into actual executable code (e.g. an ELF or EXE binary) that
-can run on a processor.  All of the previous chapters deal with rustc's
-frontend.
+So far, we've shown how the compiler can take raw source code in text format
+and transform it into MIR. We have also shown how the compiler does various
+analyses on the code to detect things like type or lifetime errors. Now, we
+will finally take the MIR and produce some executable machine code.
 
-rustc's backend does the following:
+> NOTE: This part of a compiler is often called the _backend_ the term is a bit
+> overloaded because in the compiler source, it usually refers to the "codegen
+> backend" (i.e. LLVM or Cranelift). Usually, when you see the word "backend"
+> in this part, we are refering to the "codegen backend".
+
+So what do we need to do?
 
 0. First, we need to collect the set of things to generate code for. In
    particular, we need to find out which concrete types to substitute for
@@ -23,15 +24,15 @@ rustc's backend does the following:
    collecting all the concrete types is called _monomorphization collection_.
 1. Next, we need to actually lower the MIR to a codegen IR
    (usually LLVM IR) for each concrete type we collected.
-2. Finally, we need to invoke the codegen backend (e.g. LLVM or Cranelift),
-   which runs a bunch of optimization passes, generates executable code, and
-   links together an executable binary.
+2. Finally, we need to invoke LLVM or Cranelift, which runs a bunch of
+   optimization passes, generates executable code, and links together an
+   executable binary.
 
 [codegen1]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_codegen_ssa/base/fn.codegen_crate.html
 
 The code for codegen is actually a bit complex due to a few factors:
 
-- Support for multiple backends (LLVM and Cranelift). We try to share as much
+- Support for multiple codegen backends (LLVM and Cranelift). We try to share as much
   backend code between them as possible, so a lot of it is generic over the
   codegen implementation. This means that there are often a lot of layers of
   abstraction.
@@ -48,3 +49,4 @@ crate contains code specific to LLVM codegen.
 At a very high level, the entry point is
 [`rustc_codegen_ssa::base::codegen_crate`][codegen1]. This function starts the
 process discussed in the rest of this chapter.
+

src/pat-exhaustive-checking.md

@@ -0,0 +1,7 @@
+# Pattern and Exhaustiveness Checking
+
+In Rust, pattern matching and bindings have a few very helpful properties. The
+compiler will check that bindings are irrefutable when made and that match arms
+are exhaustive.
+
+**TODO**: write this chapter.

src/syntax-intro.md

@@ -0,0 +1,8 @@
+# Syntax and the AST
+
+Working directly with source code is very inconvenient and error-prone. Thus,
+before we do anything else, we convert raw source code into an AST. It turns
+out that doing even this involves a lot of work, including lexing, parsing,
+macro expansion, name resolution, conditional compilation, feature-gate
+checking, and validation of the AST. In this chapter, we take a look at all
+of these steps.