Add mdbook-toc, markers, and documentation (#1028)

* Add mdbook-toc to travis, book.toml  and documentation

* Add toc markers

* Whitespace cleanup and some punctuation

* Addressed comments

.travis.yml

@@ -13,6 +13,7 @@ install:
 - source ~/.cargo/env || true
 - cargo install mdbook --version '^0.4.5'
 - cargo install mdbook-linkcheck --version '^0.7.2'
+- cargo install mdbook-toc --version '^0.6.1'
 script:
 - git checkout -b ci
 - git rebase origin/master

README.md

@@ -40,7 +40,7 @@ rustdocs][rustdocs].
 To build a local static HTML site, install [`mdbook`](https://github.com/rust-lang/mdBook) with:
 
 ```
-> cargo install mdbook mdbook-linkcheck
+> cargo install mdbook mdbook-linkcheck mdbook-toc
 ```
 
 and execute the following command in the root of the repository:
@@ -56,6 +56,11 @@ The build files are found in the `book` directory.
 We use `mdbook-linkcheck` to validate URLs included in our documentation.
 `linkcheck` will be run automatically when you build with the instructions in the section above.
 
+### Table of Contents
+
+We use `mdbook-toc` to auto-generate TOCs for long sections. You can invoke the preprocessor by
+including the `<!-- toc -->` marker at the place where you want the TOC.
+
 ### Pre-commit script
 
 We also test that line lengths are less than 100 columns. To test this locally,
@@ -95,7 +100,7 @@ but we leave these instructions for when we do it again in the future.
 
 7. Click on the log and Ctrl-f to get a search box in the log
 
-8. Search for rustc-dev-guide. This gets you to the place where the links are checked. It is usually ~11K lines into the log
+8. Search for rustc-dev-guide. This gets you to the place where the links are checked. It is usually ~11K lines into the log.
 
 9. Look at the links in the log near that point in the log
 

book.toml

@@ -6,6 +6,10 @@ description = "A guide to developing rustc"
 [build]
 create-missing = false
 
+[preprocessor.toc]
+command = "mdbook-toc"
+renderer = ["html"]
+
 [output.html]
 git-repository-url = "https://github.com/rust-lang/rustc-dev-guide"
 

src/about-this-guide.md

@@ -47,7 +47,7 @@ a correction!
 If you do contribute to the guide, please see the corresponding
 [subsection on writing documentation in this guide].
 
-[subsection on writing documentation in this guide]: contributing.md#contributing-to-rustc-dev-guide.
+[subsection on writing documentation in this guide]: contributing.md#contributing-to-rustc-dev-guide
 
 > “‘All conditioned things are impermanent’ — when one sees this with wisdom, one turns away from
 > suffering.” _The Dhammapada, verse 277_

src/backend/backend-agnostic.md

@@ -1,5 +1,7 @@
 # Backend Agnostic Codegen
 
+<!-- toc -->
+
 As of January 2021, `rustc_codegen_ssa` provides an abstract interface for all backends to
 implement, to allow other codegen backends (e.g. [Cranelift]).
 

src/backend/implicit-caller-location.md

@@ -1,5 +1,7 @@
 # Implicit Caller Location
 
+<!-- toc -->
+
 Approved in [RFC 2091], this feature enables the accurate reporting of caller location during panics
 initiated from functions like `Option::unwrap`, `Result::expect`, and `Index::index`. This feature
 adds the [`#[track_caller]`][attr-reference] attribute for functions, the

src/backend/monomorph.md

@@ -1,5 +1,7 @@
 # Monomorphization
 
+<!-- toc -->
+
 As you probably know, rust has a very expressive type system that has extensive
 support for generic types. But of course, assembly is not generic, so we need
 to figure out the concrete types of all the generics before the code can

src/backend/updating-llvm.md

@@ -1,5 +1,7 @@
 # Updating LLVM
 
+<!-- toc -->
+
 The Rust compiler uses LLVM as its primary codegen backend today, and naturally
 we want to at least occasionally update this dependency! Currently we do not
 have a strict policy about when to update LLVM or what it can be updated to, but

src/borrow_check/moves_and_initialization/move_paths.md

@@ -1,5 +1,7 @@
 # Move paths
 
+<!-- toc -->
+
 In reality, it's not enough to track initialization at the granularity
 of local variables. Rust also allows us to do moves and initialization
 at the field granularity:

src/borrow_check/region_inference.md

@@ -1,5 +1,7 @@
 # Region inference (NLL)
 
+<!-- toc -->
+
 The MIR-based region checking code is located in [the `rustc_mir::borrow_check`
 module][nll].
 

src/borrow_check/region_inference/constraint_propagation.md

@@ -1,5 +1,7 @@
 # Constraint propagation
 
+<!-- toc -->
+
 The main work of the region inference is **constraint propagation**,
 which is done in the [`propagate_constraints`] function.  There are
 three sorts of constraints that are used in NLL, and we'll explain how

src/borrow_check/region_inference/lifetime_parameters.md

@@ -1,5 +1,7 @@
 # Universal regions
 
+<!-- toc -->
+
 "Universal regions" is the name that the code uses to refer to "named
 lifetimes" -- e.g., lifetime parameters and `'static`. The name
 derives from the fact that such lifetimes are "universally quantified"

src/borrow_check/region_inference/member_constraints.md

@@ -1,5 +1,7 @@
 # Member constraints
 
+<!-- toc -->
+
 A member constraint `'m member of ['c_1..'c_N]` expresses that the
 region `'m` must be *equal* to some **choice regions** `'c_i` (for
 some `i`). These constraints cannot be expressed by users, but they

src/borrow_check/region_inference/placeholders_and_universes.md

@@ -1,5 +1,7 @@
 # Placeholders and universes
 
+<!-- toc -->
+
 From time to time we have to reason about regions that we can't
 concretely know. For example, consider this program:
 

src/bug-fix-procedure.md

@@ -1,4 +1,7 @@
 # Rustc Bug Fix Procedure
+
+<!-- toc -->
+
 This page defines the best practices procedure for making bug fixes or soundness
 corrections in the compiler that can cause existing code to stop compiling. This
 text is based on

src/building/bootstrapping.md

@@ -1,5 +1,7 @@
 # Bootstrapping the Compiler
 
+<!-- toc -->
+
 This subchapter is about the bootstrapping process.
 
 ## What is bootstrapping? How does it work?

src/compiler-debugging.md

@@ -1,6 +1,8 @@
 # Debugging the compiler
 [debugging]: #debugging
 
+<!-- toc -->
+
 This chapter contains a few tips to debug the compiler. These tips aim to be
 useful no matter what you are working on.  Some of the other chapters have
 advice about specific parts of the compiler (e.g. the [Queries Debugging and

src/compiler-src.md

@@ -1,11 +1,13 @@
 # High-level overview of the compiler source
 
+<!-- toc -->
+
 > **NOTE**: The structure of the repository is going through a lot of
 > transitions. In particular, we want to get to a point eventually where the
 > top-level directory has separate directories for the compiler, build-system,
 > std libs, etc, rather than one huge `src/` directory.
 >
-> As of this writing, the std libs have been moved to `library/` and the crates
+> As of January 2021, the std libs have been moved to `library/` and the crates
 > that make up the `rustc` compiler itself have been moved to `compiler/`
 
 Now that we have [seen what the compiler does](./overview.md), let's take a

src/contributing.md

@@ -3,14 +3,7 @@
 Thank you for your interest in contributing to Rust! There are many ways to
 contribute, and we appreciate all of them.
 
-* [Feature Requests](#feature-requests)
-* [Bug Reports](#bug-reports)
-* [The Build System](./building/how-to-build-and-run.md)
-* [Pull Requests](#pull-requests)
-* [Writing Documentation](#writing-documentation)
-* [Issue Triage](#issue-triage)
-* [Out-of-tree Contributions](#out-of-tree-contributions)
-* [Helpful Links and Information](#helpful-links-and-information)
+<!-- toc -->
 
 If you have questions, please make a post on [internals.rust-lang.org][internals] or
 hop on the [Rust Discord server][rust-discord] or [Rust Zulip server][rust-zulip].
@@ -423,6 +416,9 @@ Just a few things to keep in mind:
   - A link to a relevant WG, tracking issue, `rustc` rustdoc page, or similar, that may provide
     further explanation for the change process or a way to verify that the information is not
     outdated.
+- If a text grows rather long (more than a few page scrolls) or complicated (more than four
+  subsections) it might benefit from having a Table of Contents at the beginning, which you can
+  auto-generate by including the `<!-- toc -->` marker.
 
 [rdg]: https://rustc-dev-guide.rust-lang.org/
 [rdgrepo]: https://github.com/rust-lang/rustc-dev-guide

src/debugging-support-in-rustc.md

@@ -1,5 +1,7 @@
 # Debugging support in the Rust compiler
 
+<!-- toc -->
+
 This document explains the state of debugging tools support in the Rust compiler (rustc).
 The document gives an overview of debugging tools like GDB, LLDB etc. and infrastructure
 around Rust compiler to debug Rust code. If you want to learn how to debug the Rust compiler

src/diagnostics.md

@@ -1,5 +1,7 @@
 # Errors and Lints
 
+<!-- toc -->
+
 A lot of effort has been put into making `rustc` have great error messages.
 This chapter is about how to emit compile errors and lints from the compiler.
 

src/getting-started.md

@@ -1,5 +1,7 @@
 # Getting Started
 
+<!-- toc -->
+
 This documentation is _not_ intended to be comprehensive; it is meant to be a
 quick guide for the most useful things. For more information, [see this
 chapter on how to build and run the compiler](./building/how-to-build-and-run.md).

src/git.md

@@ -1,5 +1,7 @@
 # Using Git
 
+<!-- toc -->
+
 The Rust project uses [Git] to manage its source code. In order to
 contribute, you'll need some familiarity with its features so that your changes
 can be incorporated into the compiler.

src/hir.md

@@ -1,5 +1,7 @@
 # The HIR
 
+<!-- toc -->
+
 The HIR – "High-Level Intermediate Representation" – is the primary IR used
 in most of rustc. It is a compiler-friendly representation of the abstract
 syntax tree (AST) that is generated after parsing, macro expansion, and name
@@ -18,7 +20,7 @@ You can view the HIR representation of your code by passing the
 cargo rustc -- -Z unpretty=hir-tree
 ```
 
-### Out-of-band storage and the `Crate` type
+## Out-of-band storage and the `Crate` type
 
 The top-level data-structure in the HIR is the [`Crate`], which stores
 the contents of the crate currently being compiled (we only ever
@@ -66,7 +68,7 @@ the compiler a chance to observe that you accessed the data for
 
 <a name="hir-id"></a>
 
-### Identifiers in the HIR
+## Identifiers in the HIR
 
 There are a bunch of different identifiers to refer to other nodes or definitions
 in the HIR. In short:
@@ -81,7 +83,7 @@ For more detailed information, check out the [chapter on identifiers][ids].
 [`HirId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_hir/hir_id/struct.HirId.html
 [ids]: ./identifiers.md#in-the-hir
 
-### The HIR Map
+## The HIR Map
 
 Most of the time when you are working with the HIR, you will do so via
 the **HIR Map**, accessible in the tcx via [`tcx.hir()`] (and defined in
@@ -124,7 +126,7 @@ calls like [`tcx.hir().get_parent_node(n)`][get_parent_node].
 
 [get_parent_node]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/hir/map/struct.Map.html#method.get_parent_node
 
-### HIR Bodies
+## HIR Bodies
 
 A [`rustc_hir::Body`] represents some kind of executable code, such as the body
 of a function/closure or the definition of a constant. Bodies are

src/llvm-coverage-instrumentation.md

@@ -1,5 +1,7 @@
 # LLVM Source-Based Code Coverage
 
+<!-- toc -->
+
 `rustc` supports detailed source-based code and test coverage analysis
 with a command line option (`-Z instrument-coverage`) that instruments Rust
 libraries and binaries with additional instructions and data, at compile time.
@@ -32,7 +34,7 @@ Detailed instructions and examples are documented in the
 [Coverage Map]: https://llvm.org/docs/CoverageMappingFormat.html
 [unstable-book-sbcc]: https://doc.rust-lang.org/nightly/unstable-book/compiler-flags/source-based-code-coverage.html
 
-### Rust symbol mangling
+## Rust symbol mangling
 
 `-Z instrument-coverage` automatically enables Rust symbol mangling `v0` (as
 if the user specified `-Z symbol-mangling-version=v0` option when invoking

src/macro-expansion.md

@@ -1,5 +1,7 @@
 # Macro expansion
 
+<!-- toc -->
+
 > `rustc_ast`, `rustc_expand`, and `rustc_builtin_macros` are all undergoing
 > refactoring, so some of the links in this chapter may be broken.
 

src/mir/construction.md

@@ -1,5 +1,7 @@
 # THIR and MIR construction
 
+<!-- toc -->
+
 The lowering of [HIR] to [MIR] occurs for the following (probably incomplete)
 list of items:
 

src/mir/dataflow.md

@@ -1,5 +1,7 @@
 # Dataflow Analysis
 
+<!-- toc -->
+
 If you work on the MIR, you will frequently come across various flavors of
 [dataflow analysis][wiki]. `rustc` uses dataflow to find uninitialized
 variables, determine what variables are live across a generator `yield`

src/mir/index.md

@@ -1,5 +1,7 @@
 # The MIR (Mid-level IR)
 
+<!-- toc -->
+
 MIR is Rust's _Mid-level Intermediate Representation_. It is
 constructed from [HIR](../hir.html). MIR was introduced in
 [RFC 1211]. It is a radically simplified form of Rust that is used for

src/miri.md

@@ -1,5 +1,7 @@
 # Miri
 
+<!-- toc -->
+
 Miri (**MIR** **I**nterpreter) is a virtual machine for executing MIR without
 compiling to machine code. It is usually invoked via `tcx.const_eval_*` functions.
 

src/name-resolution.md

@@ -1,5 +1,7 @@
 # Name resolution
 
+<!-- toc -->
+
 In the previous chapters, we saw how the AST is built with all macros expanded.
 We saw how doing that requires doing some name resolution to resolve imports
 and macro names. In this chapter, we show how this is actually done and more.

src/overview.md

@@ -1,5 +1,7 @@
 # Overview of the Compiler
 
+<!-- toc -->
+
 This chapter is about the overall process of compiling a program -- how
 everything fits together.
 

src/panic-implementation.md

@@ -1,13 +1,15 @@
-### Panicking in rust ###
+# Panicking in rust
 
-#### Step 1: Invocation of the `panic!` macro.
+<!-- toc -->
+
+## Step 1: Invocation of the `panic!` macro.
 
 There are actually two panic macros - one defined in `core`, and one defined in `std`.
 This is due to the fact that code in `core` can panic. `core` is built before `std`,
 but we want panics to use the same machinery at runtime, whether they originate in `core`
 or `std`.
 
-##### core definition of panic!
+### core definition of panic!
 
 The `core` `panic!` macro eventually makes the following call (in `library/core/src/panicking.rs`):
 
@@ -57,7 +59,7 @@ Rust source).
 Thus, control flow will pass from core to std at runtime. This allows panics from `core`
 to go through the same infrastructure that other panics use (panic hooks, unwinding, etc)
 
-##### std implementation of panic!
+### std implementation of panic!
 
 This is where the actual panic-related logic begins. In `library/std/src/panicking.rs`,
 control passes to `rust_panic_with_hook`. This method is responsible
@@ -83,7 +85,7 @@ is suitable for passing across an FFI boundary.
 
 Finally, we call `__rust_start_panic` with this `usize`. We have now entered the panic runtime.
 
-#### Step 2: The panic runtime
+## Step 2: The panic runtime
 
 Rust provides two panic runtimes: `panic_abort` and `panic_unwind`. The user chooses
 between them at build time via their `Cargo.toml`

src/profile-guided-optimization.md

@@ -1,5 +1,7 @@
 # Profile Guided Optimization
 
+<!-- toc -->
+
 `rustc` supports doing profile-guided optimization (PGO).
 This chapter describes what PGO is and how the support for it is
 implemented in `rustc`.

src/queries/incremental-compilation-in-detail.md

@@ -1,5 +1,7 @@
 # Incremental Compilation In Detail
 
+<!-- toc -->
+
 The incremental compilation scheme is, in essence, a surprisingly
 simple extension to the overall query system. It relies on the fact that:
 

src/queries/incremental-compilation.md

@@ -1,5 +1,7 @@
 # Incremental compilation
 
+<!-- toc -->
+
 The incremental compilation scheme is, in essence, a surprisingly
 simple extension to the overall query system. We'll start by describing
 a slightly simplified variant of the real thing – the "basic algorithm" –

src/queries/profiling.md

@@ -1,4 +1,7 @@
 # Profiling Queries
+
+<!-- toc -->
+
 In an effort to support _incremental compilation_, the latest design of the Rust
 compiler consists of a _query-based_ model.
 

src/queries/query-evaluation-model-in-detail.md

@@ -1,7 +1,7 @@
-
-
 # The Query Evaluation Model in Detail
 
+<!-- toc -->
+
 This chapter provides a deeper dive into the abstract model queries are built on.
 It does not go into implementation details but tries to explain
 the underlying logic. The examples here, therefore, have been stripped down and

src/rustdoc-internals.md

@@ -1,5 +1,7 @@
 # Rustdoc internals
 
+<!-- toc -->
+
 This page describes rustdoc's passes and modes. For an overview of rustdoc,
 see [`rustdoc`](./rustdoc.md).
 

src/salsa.md

@@ -1,5 +1,7 @@
 # How Salsa works
 
+<!-- toc -->
+
 This chapter is based on the explanation given by Niko Matsakis in this
 [video](https://www.youtube.com/watch?v=_muY4HjSqVw) about
 [Salsa](https://github.com/salsa-rs/salsa). To find out more you may

src/stability.md

@@ -1,5 +1,7 @@
 # Stability attributes
 
+<!-- toc -->
+
 This section is about the stability attributes and schemes that allow stable
 APIs to use unstable APIs internally in the rustc standard library.
 

src/stabilization_guide.md

@@ -2,12 +2,9 @@
 
 Once an unstable feature has been well-tested with no outstanding
 concern, anyone may push for its stabilization. It involves the
-following steps.
+following steps:
 
-- Documentation PRs
-- Write a stabilization report
-- FCP
-- Stabilization PR
+<!-- toc -->
 
 ## Documentation PRs
 

src/test-implementation.md

@@ -1,4 +1,7 @@
-### The `#[test]` attribute
+# The `#[test]` attribute
+
+<!-- toc -->
+
 Today, rust programmers rely on a built in attribute called `#[test]`. All
 you have to do is mark a function as a test and include some asserts like so:
 
@@ -35,7 +38,7 @@ What exactly is `rustc --test` doing?
 [`rustc_ast` crate][rustc_ast]. Essentially, it's a fancy macro, that
 rewrites the crate in 3 steps:
 
-#### Step 1: Re-Exporting
+## Step 1: Re-Exporting
 
 As mentioned earlier, tests can exist inside private modules, so we need a
 way of exposing them to the main function, without breaking any existing
@@ -77,7 +80,8 @@ hand-written one, it will not share a Symbol. This technique prevents name
 collision during code generation and is the foundation of Rust's macro
 hygiene.
 
-#### Step 2: Harness Generation
+## Step 2: Harness Generation
+
 Now that our tests are accessible from the root of our crate, we need to do
 something with them. `rustc_ast` generates a module like so:
 
@@ -99,7 +103,8 @@ called [`test`][test] that is part of Rust core, that implements all of the
 runtime for testing. `test`'s interface is unstable, so the only stable way
 to interact with it is through the `#[test]` macro.
 
-#### Step 3: Test Object Generation
+## Step 3: Test Object Generation
+
 If you've written tests in Rust before, you may be familiar with some of the
 optional attributes available on test functions. For example, a test can be
 annotated with `#[should_panic]` if we expect the test to cause a panic. It
@@ -137,7 +142,8 @@ self::test::TestDescAndFn{
 Once we've constructed an array of these test objects, they're passed to the
 test runner via the harness generated in step 2.
 
-### Inspecting the generated code
+## Inspecting the generated code
+
 On nightly rust, there's an unstable flag called `unpretty` that you can use
 to print out the module source after macro expansion:
 

src/tests/adding.md

@@ -1,5 +1,7 @@
 # Adding new tests
 
+<!-- toc -->
+
 **In general, we expect every PR that fixes a bug in rustc to come
 accompanied by a regression test of some kind.** This test should fail
 in master but pass after the PR. These tests are really useful for

src/tests/intro.md

@@ -1,5 +1,7 @@
 # The compiler testing framework
 
+<!-- toc -->
+
 The Rust project runs a wide variety of different tests, orchestrated by
 the build system (`x.py test`).  The main test harness for testing the
 compiler itself is a tool called compiletest (located in the

src/tests/running.md

@@ -1,5 +1,7 @@
 # Running tests
 
+<!-- toc -->
+
 You can run the tests using `x.py`. The most basic command – which
 you will almost never want to use! – is as follows:
 

src/traits/goals-and-clauses.md

@@ -1,5 +1,7 @@
 # Goals and clauses
 
+<!-- toc -->
+
 In logic programming terms, a **goal** is something that you must
 prove and a **clause** is something that you know is true. As
 described in the [lowering to logic](./lowering-to-logic.html)

src/traits/lowering-to-logic.md

@@ -1,5 +1,7 @@
 # Lowering to logic
 
+<!-- toc -->
+
 The key observation here is that the Rust trait system is basically a
 kind of logic, and it can be mapped onto standard logical inference
 rules. We can then look for solutions to those inference rules in a

src/traits/resolution.md

@@ -1,5 +1,7 @@
 # Trait resolution (old-style)
 
+<!-- toc -->
+
 This chapter describes the general process of _trait resolution_ and points out
 some non-obvious things.
 

src/ty.md

@@ -1,5 +1,7 @@
 # The `ty` module: representing types
 
+<!-- toc -->
+
 The `ty` module defines how the Rust compiler represents types internally. It also defines the
 *typing context* (`tcx` or `TyCtxt`), which is the central data structure in the compiler.
 

src/type-inference.md

@@ -1,5 +1,7 @@
 # Type inference
 
+<!-- toc -->
+
 Type inference is the process of automatic detection of the type of an
 expression.
 

src/variance.md

@@ -1,5 +1,7 @@
 # Variance of type and lifetime parameters
 
+<!-- toc -->
+
 For a more general background on variance, see the [background] appendix.
 
 [background]: ./appendix/background.html

src/walkthrough.md

@@ -1,5 +1,7 @@
 # Walkthrough: a typical contribution
 
+<!-- toc -->
+
 There are _a lot_ of ways to contribute to the rust compiler, including fixing
 bugs, improving performance, helping design features, providing feedback on
 existing features, etc. This chapter does not claim to scratch the surface.