mirror
/
rustc-dev-guide
mirror of https://github.com/rust-lang/rustc-dev-guide.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Update rustdoc-internals.md

This commit is contained in:
Michael Howell 2023-03-13 13:37:22 -07:00 committed by Tshepang Mbambo
parent 31991ba0cb
commit ba285c2370
1 changed files with 138 additions and 84 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

222
src/rustdoc-internals.md
View File

@ -7,38 +7,61 @@ see the ["Rustdoc overview" chapter](./rustdoc.md).
## From crate to clean ## From crate to clean
In `core.rs` are two central items: the `DocContext` struct, and the `run_core` In `core.rs` are two central items: the `DocContext` struct, and the
function. The latter is where rustdoc calls out to rustc to compile a crate to `run_global_ctxt` function. The latter is where rustdoc calls out to rustc to
the point where rustdoc can take over. The former is a state container used compile a crate to the point where rustdoc can take over. The former is a state
when crawling through a crate to gather its documentation. container used when crawling through a crate to gather its documentation.
The main process of crate crawling is done in `clean/mod.rs` through several The main process of crate crawling is done in `clean/mod.rs` through several
implementations of the `Clean` trait defined within. This is a conversion functions with names that start with `clean_`. Each function accepts an `hir`
trait, which defines one method: or `ty` data structure, and outputs a `clean` structure used by rustdoc. For
example, this function for converting lifetimes:
```rust,ignore ```rust,ignore
pub trait Clean<T> { fn clean_lifetime<'tcx>(lifetime: &hir::Lifetime, cx: &mut DocContext<'tcx>) -> Lifetime {
fn clean(&self, cx: &DocContext) -> T; let def = cx.tcx.named_bound_var(lifetime.hir_id);
if let Some(
rbv::ResolvedArg::EarlyBound(node_id)
| rbv::ResolvedArg::LateBound(_, _, node_id)
| rbv::ResolvedArg::Free(_, node_id),
) = def
{
if let Some(lt) = cx.substs.get(&node_id).and_then(|p| p.as_lt()).cloned() {
return lt;
}
}
Lifetime(lifetime.ident.name)
} }
``` ```
`clean/mod.rs` also defines the types for the "cleaned" AST used later on to `clean/mod.rs` also defines the types for the "cleaned" AST used later on to
render documentation pages. Each usually accompanies an implementation of render documentation pages. Each usually accompanies a `clean` function
`Clean` that takes some AST or HIR type from rustc and converts it into the that takes some AST or HIR type from rustc and converts it into the
appropriate "cleaned" type. "Big" items like modules or associated items may appropriate "cleaned" type. "Big" items like modules or associated items may
have some extra processing in its `Clean` implementation, but for the most part have some extra processing in its `clean` function, but for the most part
these impls are straightforward conversions. The "entry point" to this module these impls are straightforward conversions. The "entry point" to this module
is the `impl Clean<Crate> for visit_ast::RustdocVisitor`, which is called by is `clean::krate`, which is called by
`run_core` above. `run_global_ctxt` above.
You see, I actually lied a little earlier: There's another AST transformation The first step in `clean::krate` is to invoke `visit_ast::RustdocVisitor` to
that happens before the events in `clean/mod.rs`. In `visit_ast.rs` is the process the module tree into an intermediate `visit_ast::Module`. This is the
type `RustdocVisitor`, which *actually* crawls a `rustc_hir::Crate` to get the first step that actually crawls the `rustc_hir::Crate`, normalizing various aspects
intermediate representation, defined in `doctree.rs`. This pass is mainly to of name resolution, such as:
get a few intermediate wrappers around the HIR types and to process visibility
and inlining. This is where `#[doc(inline)]`, `#[doc(no_inline)]`, and * showing `#[macro_export]`-ed macros at the crate root, regardless of where
`#[doc(hidden)]` are processed, as well as the logic for whether a `pub use` they're defined
should get the full page or a "Reexport" line in the module page. * inlining public `use` exports of private items, or showing a "Reexport"
line in the module page
* inlining items with `#[doc(hidden)]` if the base item is hidden but the
reexport is not
* handling `#[doc(inline)]` and `#[doc(no_inline)]`
* handling import globs and cycles, so there are no duplicates or infinite
directory trees
After this step, `clean::krate` invokes `clean_doc_module`, which actually
converts the HIR items to the cleaned AST. This is also the step where cross-
crate inlining is performed, which requires converting `rustc_middle` data
structures into the cleaned AST instead.
The other major thing that happens in `clean/mod.rs` is the collection of doc The other major thing that happens in `clean/mod.rs` is the collection of doc
comments and `#[doc=""]` attributes into a separate field of the Attributes comments and `#[doc=""]` attributes into a separate field of the Attributes
@ -48,41 +71,28 @@ easier to collect this documentation later in the process.
The primary output of this process is a `clean::Crate` with a tree of Items The primary output of this process is a `clean::Crate` with a tree of Items
which describe the publicly-documentable items in the target crate. which describe the publicly-documentable items in the target crate.
### Hot potato ### Passes anything but a gas station
(alternate title: [hot potato](https://www.youtube.com/watch?v=WNFBIt5HxdY))
Before moving on to the next major step, a few important "passes" occur over Before moving on to the next major step, a few important "passes" occur over
the documentation. These do things like combine the separate "attributes" into the cleaned AST. Several of these passes are lints and reports, but some of
a single string to make the document easier on the markdown parser, them mutate or generate new items.
or drop items that are not public or deliberately hidden with `#[doc(hidden)]`.
These are all implemented in the `passes/` directory, one file per pass. These are all implemented in the `passes/` directory, one file per pass.
By default, all of these passes are run on a crate, but the ones By default, all of these passes are run on a crate, but the ones
regarding dropping private/hidden items can be bypassed by passing regarding dropping private/hidden items can be bypassed by passing
`--document-private-items` to rustdoc. Note that unlike the previous set of AST `--document-private-items` to rustdoc. Note that unlike the previous set of AST
transformations, the passes are run on the _cleaned_ crate. transformations, the passes are run on the _cleaned_ crate.
(Strictly speaking, you can fine-tune the passes run and even add your own, but Here is the list of passes as of <!-- date-check --> March 2023:
[we're trying to deprecate that][44136]. If you need finer-grain control over
these passes, please let us know!)
[44136]: https://github.com/rust-lang/rust/issues/44136
Here is the list of passes as of <!-- date-check --> November 2022:
- `calculate-doc-coverage` calculates information used for the `--show-coverage` - `calculate-doc-coverage` calculates information used for the `--show-coverage`
flag. flag.
- `check-bare-urls` detects links that are not linkified, e.g., in Markdown such as - `check-doc-test-visibility` runs doctest visibility–related lints. This pass
`Go to https://example.com/.` It suggests wrapping the link with angle brackets: runs before `strip-private`, which is why it needs to be separate from
`Go to <https://example.com/>.` to linkify it. This is the code behind the <!-- `run-lints`.
date: 2022-05 --> `rustdoc::bare_urls` lint.
- `check-code-block-syntax` validates syntax inside Rust code blocks
(<code>```rust</code>)
- `check-doc-test-visibility` runs doctest visibility–related lints.
- `check-invalid-html-tags` detects invalid HTML (like an unclosed `<span>`)
in doc comments.
- `collect-intra-doc-links` resolves [intra-doc links](https://doc.rust-lang.org/nightly/rustdoc/write-documentation/linking-to-items-by-name.html). - `collect-intra-doc-links` resolves [intra-doc links](https://doc.rust-lang.org/nightly/rustdoc/write-documentation/linking-to-items-by-name.html).
@ -92,44 +102,66 @@ Here is the list of passes as of <!-- date-check --> November 2022:
- `propagate-doc-cfg` propagates `#[doc(cfg(...))]` to child items. - `propagate-doc-cfg` propagates `#[doc(cfg(...))]` to child items.
- `run-lints` runs some of rustdoc's lints, defind in `passes/lint`. This is
the last pass to run.
- `bare_urls` detects links that are not linkified, e.g., in Markdown such as
`Go to https://example.com/.` It suggests wrapping the link with angle brackets:
`Go to <https://example.com/>.` to linkify it. This is the code behind the <!--
date: 2022-05 --> `rustdoc::bare_urls` lint.
- `check_code_block_syntax` validates syntax inside Rust code blocks
(<code>```rust</code>)
- `html_tags` detects invalid HTML (like an unclosed `<span>`)
in doc comments.
- `strip-hidden` and `strip-private` strip all `doc(hidden)` and private items
from the output. `strip-private` implies `strip-priv-imports`. Basically, the
goal is to remove items that are not relevant for public documentation. This
pass is skipped when `--document-hidden-items` is passed.
- `strip-priv-imports` strips all private import statements (`use`, `extern - `strip-priv-imports` strips all private import statements (`use`, `extern
crate`) from a crate. This is necessary because rustdoc will handle *public* crate`) from a crate. This is necessary because rustdoc will handle *public*
imports by either inlining the item's documentation to the module or creating imports by either inlining the item's documentation to the module or creating
a "Reexports" section with the import in it. The pass ensures that all of a "Reexports" section with the import in it. The pass ensures that all of
these imports are actually relevant to documentation. these imports are actually relevant to documentation. It is technically
only run when `--document-private-items` is passed, but `strip-private`
accomplishes the same thing.
- `strip-hidden` and `strip-private` strip all `doc(hidden)` and private items - `strip-private` strips all private items from a crate which cannot be seen
from the output. `strip-private` implies `strip-priv-imports`. Basically, the externally. This pass is skipped when `--document-private-items` is passed.
goal is to remove items that are not relevant for public documentation.
There is also a `stripper` module in `passes/`, but it is a collection of There is also a `stripper` module in `passes/`, but it is a collection of
utility functions for the `strip-*` passes and is not a pass itself. utility functions for the `strip-*` passes and is not a pass itself.
## From clean to crate ## From clean to HTML
This is where the "second phase" in rustdoc begins. This phase primarily lives This is where the "second phase" in rustdoc begins. This phase primarily lives
in the `html/` folder, and it all starts with `run()` in `html/render.rs`. This in the `formats/` and `html/` folders, and it all starts with
code is responsible for setting up the `Context`, `SharedContext`, and `Cache` `formats::run_format`. This code is responsible for setting up a type that
which are used during rendering, copying out the static files which live in `impl FormatRenderer`, which for HTML is [`Context`].
every rendered set of documentation (things like the fonts, CSS, and JavaScript
that live in `html/static/`), creating the search index, and printing out the
source code rendering, before beginning the process of rendering all the
documentation for the crate.
Several functions implemented directly on `Context` take the `clean::Crate` and This structure contains methods that get called by `run_format` to drive the
set up some state between rendering items or recursing on a module's child doc rendering, which includes:
items. From here the "page rendering" begins, via an enormous `write!()` call
in `html/layout.rs`. The parts that actually generate HTML from the items and * `init` generates `static.files`, as well as search index and `src/`
documentation occurs within a series of `std::fmt::Display` implementations and * `item` generates the item HTML files themselves
functions that pass around a `&mut std::fmt::Formatter`. The top-level * `after_krate` generates other global resources like `all.html`
implementation that writes out the page body is the `impl<'a> fmt::Display for
Item<'a>` in `html/render.rs`, which switches out to one of several `item_*` In `item`, the "page rendering" occurs, via a mixture of [Askama] templates
functions based on the kind of `Item` being rendered. and manual `write!()` calls, starting in `html/layout.rs`. The parts that have
not been converted to templates occur within a series of `std::fmt::Display`
implementations and functions that pass around a `&mut std::fmt::Formatter`.
The parts that actually generate HTML from the items and documentation start
with `print_item` defined in `html/render/print_item.rs`, which switches out
to one of several `item_*` functions based on kind of `Item` being rendered.
Depending on what kind of rendering code you're looking for, you'll probably Depending on what kind of rendering code you're looking for, you'll probably
find it either in `html/render.rs` for major items like "what sections should I find it either in `html/render/mod.rs` for major items like "what sections
print for a struct page" or `html/format.rs` for smaller component pieces like should I print for a struct page" or `html/format/mod.rs` for smaller
"how should I print a where clause as part of some other item". component pieces like "how should I print a where clause as part of some other item".
Whenever rustdoc comes across an item that should print hand-written Whenever rustdoc comes across an item that should print hand-written
documentation alongside, it calls out to `html/markdown.rs` which interfaces documentation alongside, it calls out to `html/markdown.rs` which interfaces
@ -148,23 +180,45 @@ to us"][video])
[video]: https://www.youtube.com/watch?v=hOLAGYmUQV0 [video]: https://www.youtube.com/watch?v=hOLAGYmUQV0
It's important to note that the AST cleaning can ask the compiler for It's important to note that rustdoc can ask the compiler for type information
information (crucially, `DocContext` contains a `TyCtxt`), but page rendering directly, even during HTML generation. This [didn't used to be the case], and
cannot. The `clean::Crate` created within `run_core` is passed outside the a lot of rustdoc's architecture was designed around not doing that, but a
compiler context before being handed to `html::render::run`. This means that a `TyCtxt` is now passed to `formats::renderer::run_format`, which is used to
lot of the "supplementary data" that isn't immediately available inside an run generation for both HTML and the (unstable as of March 2023) JSON format.
item's definition, like which trait is the `Deref` trait used by the language,
needs to be collected during cleaning, stored in the `DocContext`, and passed
along to the `SharedContext` during HTML rendering. This manifests as a bunch
of shared state, context variables, and `RefCell`s.
Also of note is that some items that come from "asking the compiler" don't go [didn't used to be the case]: https://github.com/rust-lang/rust/pull/80090
directly into the `DocContext` - for example, when loading items from a foreign
crate, rustdoc will ask about trait implementations and generate new `Item`s This change has allowed other changes to remove data from the "clean" AST
for the impls based on that information. This goes directly into the returned that can be easily derived from `TyCtxt` queries, and we'll usually accept
`Crate` rather than roundabout through the `DocContext`. This way, these PRs that remove fields from "clean" (it's been soft-deprecated), but this
implementations can be collected alongside the others, right before rendering is complicated from two other constraints that rustdoc runs under:
the HTML.
* Docs can be generated for crates that don't actually pass type checking.
This is used for generating docs that cover mutually-exclusive platform
configurations, such as `libstd` having a single package of docs that
cover all supported operating systems. This means rustdoc has to be able
to generate docs from HIR.
* Docs can inline across crates. Since crate metadata doesn't contain HIR,
it must be possible to generate inlined docs from the `rustc_middle` data.
The "clean" AST acts as a common output format for both input formats. There
is also some data in clean that doesn't correspond directly to HIR, such as
synthetic `impl`s for auto traits and blanket `impl`s generated by the
`collect-trait-impls` pass.
Some additional data is stored in
`html::render::context::{Context, SharedContext}`. These two types serve as
ways to segregate rustdoc's data for an eventual future with multithreaded doc
generation, as well as just keeping things organized:
* [`Context`] stores data used for generating the current page, such as its
path, a list of HTML IDs that have been used (to avoid duplicate `id=""`),
and the pointer to `SharedContext`.
* [`SharedContext`] stores data that does not vary by page, such as the `tcx`
pointer, and a list of all types.
[`Context`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/html/render/context/struct.Context.html
[`SharedContext`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/html/render/context/struct.SharedContext.html
## Other tricks up its sleeve ## Other tricks up its sleeve