mirror
/
rustc-dev-guide
mirror of https://github.com/rust-lang/rustc-dev-guide.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

reformat basically the whole rustdoc.md

This commit is contained in:
Mark Mansi 2018-03-26 10:46:39 -05:00 committed by Who? Me?!
parent 9e6cdf3de3
commit 066ba9ef82
1 changed files with 153 additions and 122 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

275
src/rustdoc.md
View File

@ -1,7 +1,8 @@
# The walking tour of rustdoc # The walking tour of rustdoc
Rustdoc actually uses the rustc internals directly. It lives in-tree with the compiler and standard Rustdoc actually uses the rustc internals directly. It lives in-tree with the
library. This chapter is about how it works. (A new implementation is also [under way], though). compiler and standard library. This chapter is about how it works. (A new
implementation is also [under way], though).
[under way]: https://github.com/steveklabnik/rustdoc [under way]: https://github.com/steveklabnik/rustdoc
@ -17,50 +18,57 @@ and [queries] are discussed in the linked chapters.
`librustdoc` performs two major steps after that to render a set of `librustdoc` performs two major steps after that to render a set of
documentation: documentation:
* "Clean" the AST into a form that's more suited to creating documentation (and slightly more * "Clean" the AST into a form that's more suited to creating documentation (and
resistant to churn in the compiler). slightly more resistant to churn in the compiler).
* Use this cleaned AST to render a crate's documentation, one page at a time. * Use this cleaned AST to render a crate's documentation, one page at a time.
Naturally, there's more than just this, and those descriptions simplify out lots of details, but Naturally, there's more than just this, and those descriptions simplify out
that's the high-level overview. lots of details, but that's the high-level overview.
(Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated using the project in (Side note: `librustdoc` is a library crate! The `rustdoc` binary is crated
[`src/tools/rustdoc`][bin]. Note that literally all that does is call the `main()` that's in this crate's using the project in [`src/tools/rustdoc`][bin]. Note that literally all that
`lib.rs`, though.) does is call the `main()` that's in this crate's `lib.rs`, though.)
[bin]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc [bin]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc
## Cheat sheet ## Cheat sheet
* Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable rustdoc you can run on * Use `x.py build --stage 1 src/libstd src/tools/rustdoc` to make a useable
other projects. rustdoc you can run on other projects.
* Add `src/libtest` to be able to use `rustdoc --test`. * Add `src/libtest` to be able to use `rustdoc --test`.
* If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1` previously, then * If you've used `rustup toolchain link local /path/to/build/$TARGET/stage1`
after the previous build command, `cargo +local doc` will Just Work. previously, then after the previous build command, `cargo +local doc` will
* Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the standard library docs. Just Work.
* The completed docs will be available in `build/$TARGET/doc/std`, though the bundle is meant to * Use `x.py doc --stage 1 src/libstd` to use this rustdoc to generate the
be used as though you would copy out the `doc` folder to a web server, since that's where the standard library docs.
CSS/JS and landing page are. * The completed docs will be available in `build/$TARGET/doc/std`, though the
* Most of the HTML printing code is in `html/format.rs` and `html/render.rs`. It's in a bunch of bundle is meant to be used as though you would copy out the `doc` folder to
`fmt::Display` implementations and supplementary functions. a web server, since that's where the CSS/JS and landing page are.
* The types that got `Display` impls above are defined in `clean/mod.rs`, right next to the custom * Most of the HTML printing code is in `html/format.rs` and `html/render.rs`.
`Clean` trait used to process them out of the rustc HIR. It's in a bunch of `fmt::Display` implementations and supplementary
functions.
* The types that got `Display` impls above are defined in `clean/mod.rs`, right
next to the custom `Clean` trait used to process them out of the rustc HIR.
* The bits specific to using rustdoc as a test harness are in `test.rs`. * The bits specific to using rustdoc as a test harness are in `test.rs`.
* The Markdown renderer is loaded up in `html/markdown.rs`, including functions for extracting * The Markdown renderer is loaded up in `html/markdown.rs`, including functions
doctests from a given block of Markdown. for extracting doctests from a given block of Markdown.
* The tests on rustdoc *output* are located in `src/test/rustdoc`, where they're handled by the test * The tests on rustdoc *output* are located in `src/test/rustdoc`, where
runner of rustbuild and the supplementary script `src/etc/htmldocck.py`. they're handled by the test runner of rustbuild and the supplementary script
* Tests on search index generation are located in `src/test/rustdoc-js`, as a series of JavaScript `src/etc/htmldocck.py`.
files that encode queries on the standard library search index and expected results. * Tests on search index generation are located in `src/test/rustdoc-js`, as a
series of JavaScript files that encode queries on the standard library search
index and expected results.
## From crate to clean ## From crate to clean
In `core.rs` are two central items: the `DocContext` struct, and the `run_core` function. The latter In `core.rs` are two central items: the `DocContext` struct, and the `run_core`
is where rustdoc calls out to rustc to compile a crate to the point where rustdoc can take over. The function. The latter is where rustdoc calls out to rustc to compile a crate to
former is a state container used when crawling through a crate to gather its documentation. the point where rustdoc can take over. The former is a state container used
when crawling through a crate to gather its documentation.
The main process of crate crawling is done in `clean/mod.rs` through several implementations of the The main process of crate crawling is done in `clean/mod.rs` through several
`Clean` trait defined within. This is a conversion trait, which defines one method: implementations of the `Clean` trait defined within. This is a conversion
trait, which defines one method:
```rust ```rust
pub trait Clean<T> { pub trait Clean<T> {
@ -68,40 +76,47 @@ pub trait Clean<T> {
} }
``` ```
`clean/mod.rs` also defines the types for the "cleaned" AST used later on to render documentation `clean/mod.rs` also defines the types for the "cleaned" AST used later on to
pages. Each usually accompanies an implementation of `Clean` that takes some AST or HIR type from render documentation pages. Each usually accompanies an implementation of
rustc and converts it into the appropriate "cleaned" type. "Big" items like modules or associated `Clean` that takes some AST or HIR type from rustc and converts it into the
items may have some extra processing in its `Clean` implementation, but for the most part these appropriate "cleaned" type. "Big" items like modules or associated items may
impls are straightforward conversions. The "entry point" to this module is the `impl Clean<Crate> have some extra processing in its `Clean` implementation, but for the most part
for visit_ast::RustdocVisitor`, which is called by `run_core` above. these impls are straightforward conversions. The "entry point" to this module
is the `impl Clean<Crate> for visit_ast::RustdocVisitor`, which is called by
`run_core` above.
You see, I actually lied a little earlier: There's another AST transformation that happens before You see, I actually lied a little earlier: There's another AST transformation
the events in `clean/mod.rs`. In `visit_ast.rs` is the type `RustdocVisitor`, which *actually* that happens before the events in `clean/mod.rs`. In `visit_ast.rs` is the
crawls a `hir::Crate` to get the first intermediate representation, defined in `doctree.rs`. This type `RustdocVisitor`, which *actually* crawls a `hir::Crate` to get the first
pass is mainly to get a few intermediate wrappers around the HIR types and to process visibility intermediate representation, defined in `doctree.rs`. This pass is mainly to
and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and `#[doc(hidden)]` are get a few intermediate wrappers around the HIR types and to process visibility
processed, as well as the logic for whether a `pub use` should get the full page or a "Reexport" and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and
line in the module page. `#[doc(hidden)]` are processed, as well as the logic for whether a `pub use`
should get the full page or a "Reexport" line in the module page.
The other major thing that happens in `clean/mod.rs` is the collection of doc comments and The other major thing that happens in `clean/mod.rs` is the collection of doc
`#[doc=""]` attributes into a separate field of the Attributes struct, present on anything that gets comments and `#[doc=""]` attributes into a separate field of the Attributes
hand-written documentation. This makes it easier to collect this documentation later in the process. struct, present on anything that gets hand-written documentation. This makes it
easier to collect this documentation later in the process.
The primary output of this process is a `clean::Crate` with a tree of Items which describe the The primary output of this process is a `clean::Crate` with a tree of Items
publicly-documentable items in the target crate. which describe the publicly-documentable items in the target crate.
### Hot potato ### Hot potato
Before moving on to the next major step, a few important "passes" occur over the documentation. Before moving on to the next major step, a few important "passes" occur over
These do things like combine the separate "attributes" into a single string and strip leading the documentation. These do things like combine the separate "attributes" into
whitespace to make the document easier on the markdown parser, or drop items that are not public or a single string and strip leading whitespace to make the document easier on the
deliberately hidden with `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one markdown parser, or drop items that are not public or deliberately hidden with
file per pass. By default, all of these passes are run on a crate, but the ones regarding dropping `#[doc(hidden)]`. These are all implemented in the `passes/` directory, one
private/hidden items can be bypassed by passing `--document-private-items` to rustdoc. Note that file per pass. By default, all of these passes are run on a crate, but the ones
unlike the previous set of AST transformations, the passes happen on the _cleaned_ crate. regarding dropping private/hidden items can be bypassed by passing
`--document-private-items` to rustdoc. Note that unlike the previous set of AST
transformations, the passes happen on the _cleaned_ crate.
(Strictly speaking, you can fine-tune the passes run and even add your own, but [we're trying to (Strictly speaking, you can fine-tune the passes run and even add your own, but
deprecate that][44136]. If you need finer-grain control over these passes, please let us know!) [we're trying to deprecate that][44136]. If you need finer-grain control over
these passes, please let us know!)
[44136]: https://github.com/rust-lang/rust/issues/44136 [44136]: https://github.com/rust-lang/rust/issues/44136
@ -116,9 +131,9 @@ Here is current (as of this writing) list of passes:
markdown to like it. This is necessary because the convention for writing markdown to like it. This is necessary because the convention for writing
documentation is to provide a space between the `///` or `//!` marker and the documentation is to provide a space between the `///` or `//!` marker and the
text, and stripping that leading space will make the text easier to parse by text, and stripping that leading space will make the text easier to parse by
the Markdown parser. (In the past, the markdown parser used was not Commonmark- the Markdown parser. (In the past, the markdown parser used was not
compliant, which caused annoyances with extra whitespace but this seems to be Commonmark- compliant, which caused annoyances with extra whitespace but this
less of an issue today.) seems to be less of an issue today.)
- `strip-priv-imports` strips all private import statements (`use`, `extern - `strip-priv-imports` strips all private import statements (`use`, `extern
crate`) from a crate. This is necessary because rustdoc will handle *public* crate`) from a crate. This is necessary because rustdoc will handle *public*
imports by either inlining the item's documentation to the module or creating imports by either inlining the item's documentation to the module or creating
@ -130,85 +145,101 @@ Here is current (as of this writing) list of passes:
## From clean to crate ## From clean to crate
This is where the "second phase" in rustdoc begins. This phase primarily lives in the `html/` This is where the "second phase" in rustdoc begins. This phase primarily lives
folder, and it all starts with `run()` in `html/render.rs`. This code is responsible for setting up in the `html/` folder, and it all starts with `run()` in `html/render.rs`. This
the `Context`, `SharedContext`, and `Cache` which are used during rendering, copying out the static code is responsible for setting up the `Context`, `SharedContext`, and `Cache`
files which live in every rendered set of documentation (things like the fonts, CSS, and JavaScript which are used during rendering, copying out the static files which live in
that live in `html/static/`), creating the search index, and printing out the source code rendering, every rendered set of documentation (things like the fonts, CSS, and JavaScript
before beginning the process of rendering all the documentation for the crate. that live in `html/static/`), creating the search index, and printing out the
source code rendering, before beginning the process of rendering all the
documentation for the crate.
Several functions implemented directly on `Context` take the `clean::Crate` and set up some state Several functions implemented directly on `Context` take the `clean::Crate` and
between rendering items or recursing on a module's child items. From here the "page rendering" set up some state between rendering items or recursing on a module's child
begins, via an enormous `write!()` call in `html/layout.rs`. The parts that actually generate HTML items. From here the "page rendering" begins, via an enormous `write!()` call
from the items and documentation occurs within a series of `std::fmt::Display` implementations and in `html/layout.rs`. The parts that actually generate HTML from the items and
functions that pass around a `&mut std::fmt::Formatter`. The top-level implementation that writes documentation occurs within a series of `std::fmt::Display` implementations and
out the page body is the `impl<'a> fmt::Display for Item<'a>` in `html/render.rs`, which switches functions that pass around a `&mut std::fmt::Formatter`. The top-level
out to one of several `item_*` functions based on the kind of `Item` being rendered. implementation that writes out the page body is the `impl<'a> fmt::Display for
Item<'a>` in `html/render.rs`, which switches out to one of several `item_*`
functions based on the kind of `Item` being rendered.
Depending on what kind of rendering code you're looking for, you'll probably find it either in Depending on what kind of rendering code you're looking for, you'll probably
`html/render.rs` for major items like "what sections should I print for a struct page" or find it either in `html/render.rs` for major items like "what sections should I
`html/format.rs` for smaller component pieces like "how should I print a where clause as part of print for a struct page" or `html/format.rs` for smaller component pieces like
some other item". "how should I print a where clause as part of some other item".
Whenever rustdoc comes across an item that should print hand-written documentation alongside, it Whenever rustdoc comes across an item that should print hand-written
calls out to `html/markdown.rs` which interfaces with the Markdown parser. This is exposed as a documentation alongside, it calls out to `html/markdown.rs` which interfaces
series of types that wrap a string of Markdown, and implement `fmt::Display` to emit HTML text. It with the Markdown parser. This is exposed as a series of types that wrap a
takes special care to enable certain features like footnotes and tables and add syntax highlighting string of Markdown, and implement `fmt::Display` to emit HTML text. It takes
to Rust code blocks (via `html/highlight.rs`) before running the Markdown parser. There's also a special care to enable certain features like footnotes and tables and add
function in here (`find_testable_code`) that specifically scans for Rust code blocks so the syntax highlighting to Rust code blocks (via `html/highlight.rs`) before
running the Markdown parser. There's also a function in here
(`find_testable_code`) that specifically scans for Rust code blocks so the
test-runner code can find all the doctests in the crate. test-runner code can find all the doctests in the crate.
### From soup to nuts ### From soup to nuts
(alternate title: ["An unbroken thread that stretches from those first `Cell`s to us"][video]) (alternate title: ["An unbroken thread that stretches from those first `Cell`s
to us"][video])
[video]: https://www.youtube.com/watch?v=hOLAGYmUQV0 [video]: https://www.youtube.com/watch?v=hOLAGYmUQV0
It's important to note that the AST cleaning can ask the compiler for information (crucially, It's important to note that the AST cleaning can ask the compiler for
`DocContext` contains a `TyCtxt`), but page rendering cannot. The `clean::Crate` created within information (crucially, `DocContext` contains a `TyCtxt`), but page rendering
`run_core` is passed outside the compiler context before being handed to `html::render::run`. This cannot. The `clean::Crate` created within `run_core` is passed outside the
means that a lot of the "supplementary data" that isn't immediately available inside an item's compiler context before being handed to `html::render::run`. This means that a
definition, like which trait is the `Deref` trait used by the language, needs to be collected during lot of the "supplementary data" that isn't immediately available inside an
cleaning, stored in the `DocContext`, and passed along to the `SharedContext` during HTML rendering. item's definition, like which trait is the `Deref` trait used by the language,
This manifests as a bunch of shared state, context variables, and `RefCell`s. needs to be collected during cleaning, stored in the `DocContext`, and passed
along to the `SharedContext` during HTML rendering. This manifests as a bunch
of shared state, context variables, and `RefCell`s.
Also of note is that some items that come from "asking the compiler" don't go directly into the Also of note is that some items that come from "asking the compiler" don't go
`DocContext` - for example, when loading items from a foreign crate, rustdoc will ask about trait directly into the `DocContext` - for example, when loading items from a foreign
implementations and generate new `Item`s for the impls based on that information. This goes directly crate, rustdoc will ask about trait implementations and generate new `Item`s
into the returned `Crate` rather than roundabout through the `DocContext`. This way, these for the impls based on that information. This goes directly into the returned
implementations can be collected alongside the others, right before rendering the HTML. `Crate` rather than roundabout through the `DocContext`. This way, these
implementations can be collected alongside the others, right before rendering
the HTML.
## Other tricks up its sleeve ## Other tricks up its sleeve
All this describes the process for generating HTML documentation from a Rust crate, but there are All this describes the process for generating HTML documentation from a Rust
couple other major modes that rustdoc runs in. It can also be run on a standalone Markdown file, or crate, but there are couple other major modes that rustdoc runs in. It can also
it can run doctests on Rust code or standalone Markdown files. For the former, it shortcuts straight be run on a standalone Markdown file, or it can run doctests on Rust code or
to `html/markdown.rs`, optionally including a mode which inserts a Table of Contents to the output standalone Markdown files. For the former, it shortcuts straight to
HTML. `html/markdown.rs`, optionally including a mode which inserts a Table of
Contents to the output HTML.
For the latter, rustdoc runs a similar partial-compilation to get relevant documentation in For the latter, rustdoc runs a similar partial-compilation to get relevant
`test.rs`, but instead of going through the full clean and render process, it runs a much simpler documentation in `test.rs`, but instead of going through the full clean and
crate walk to grab *just* the hand-written documentation. Combined with the aforementioned render process, it runs a much simpler crate walk to grab *just* the
"`find_testable_code`" in `html/markdown.rs`, it builds up a collection of tests to run before hand-written documentation. Combined with the aforementioned
handing them off to the libtest test runner. One notable location in `test.rs` is the function "`find_testable_code`" in `html/markdown.rs`, it builds up a collection of
`make_test`, which is where hand-written doctests get transformed into something that can be tests to run before handing them off to the libtest test runner. One notable
executed. location in `test.rs` is the function `make_test`, which is where hand-written
doctests get transformed into something that can be executed.
Some extra reading about `make_test` can be found Some extra reading about `make_test` can be found
[here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/). [here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/).
## Dotting i's and crossing t's ## Dotting i's and crossing t's
So that's rustdoc's code in a nutshell, but there's more things in the repo that deal with it. Since So that's rustdoc's code in a nutshell, but there's more things in the repo
we have the full `compiletest` suite at hand, there's a set of tests in `src/test/rustdoc` that make that deal with it. Since we have the full `compiletest` suite at hand, there's
sure the final HTML is what we expect in various situations. These tests also use a supplementary a set of tests in `src/test/rustdoc` that make sure the final HTML is what we
script, `src/etc/htmldocck.py`, that allows it to look through the final HTML using XPath notation expect in various situations. These tests also use a supplementary script,
to get a precise look at the output. The full description of all the commands available to rustdoc `src/etc/htmldocck.py`, that allows it to look through the final HTML using
tests is in `htmldocck.py`. XPath notation to get a precise look at the output. The full description of all
the commands available to rustdoc tests is in `htmldocck.py`.
In addition, there are separate tests for the search index and rustdoc's ability to query it. The In addition, there are separate tests for the search index and rustdoc's
files in `src/test/rustdoc-js` each contain a different search query and the expected results, ability to query it. The files in `src/test/rustdoc-js` each contain a
broken out by search tab. These files are processed by a script in `src/tools/rustdoc-js` and the different search query and the expected results, broken out by search tab.
Node.js runtime. These tests don't have as thorough of a writeup, but a broad example that features These files are processed by a script in `src/tools/rustdoc-js` and the Node.js
results in all tabs can be found in `basic.js`. The basic idea is that you match a given `QUERY` runtime. These tests don't have as thorough of a writeup, but a broad example
with a set of `EXPECTED` results, complete with the full item path of each item. that features results in all tabs can be found in `basic.js`. The basic idea is
that you match a given `QUERY` with a set of `EXPECTED` results, complete with
the full item path of each item.