mirror
/
rustc-dev-guide
mirror of https://github.com/rust-lang/rustc-dev-guide.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

rustdoc: Further improve chapters and sections on testing

This commit is contained in:
León Orell Valerian Liehr 2025-03-20 09:11:45 +01:00
parent 16525780ff
commit f5e1a73fb5
No known key found for this signature in database
GPG Key ID: D17A07215F68E713
8 changed files with 201 additions and 128 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

2
src/SUMMARY.md
View File

@ -101,6 +101,8 @@
- [Rustdoc internals](./rustdoc-internals.md)
- [Search](./rustdoc-internals/search.md)
- [The `rustdoc` test suite](./rustdoc-internals/rustdoc-test-suite.md)
- [The `rustdoc-gui` test suite](./rustdoc-internals/rustdoc-gui-test-suite.md)
- [The `rustdoc-json` test suite](./rustdoc-internals/rustdoc-json-test-suite.md)
- [Autodiff internals](./autodiff/internals.md)
- [Installation](./autodiff/installation.md)
- [How to debug](./autodiff/debugging.md)

29
src/rustdoc-internals.md
View File

@ -270,35 +270,6 @@ in `test.rs` is the function `make_test`, which is where hand-written
Some extra reading about `make_test` can be found
[here](https://quietmisdreavus.net/code/2018/02/23/how-the-doctests-get-made/).
## Dotting i's And Crossing t's
So that's `rustdoc`'s code in a nutshell, but there's more things in the
compiler that deal with it. Since we have the full `compiletest` suite at hand,
there's a set of tests in `tests/rustdoc` that make sure the final `HTML` is
what we expect in various situations. These tests also use a supplementary
script, `src/etc/htmldocck.py`, that allows it to look through the final `HTML`
using `XPath` notation to get a precise look at the output. The full
description of all the commands available to `rustdoc` tests (e.g. [`@has`] and
[`@matches`]) is in [`htmldocck.py`].
To use multiple crates in a `rustdoc` test, add `//@ aux-build:filename.rs`
to the top of the test file. `filename.rs` should be placed in an `auxiliary`
directory relative to the test file with the comment. If you need to build
docs for the auxiliary file, use `//@ build-aux-docs`.
In addition, there are separate tests for the search index and `rustdoc`'s
ability to query it. The files in `tests/rustdoc-js` each contain a
different search query and the expected results, broken out by search tab.
These files are processed by a script in `src/tools/rustdoc-js` and the `Node.js`
runtime. These tests don't have as thorough of a writeup, but a broad example
that features results in all tabs can be found in `basic.js`. The basic idea is
that you match a given `QUERY` with a set of `EXPECTED` results, complete with
the full item path of each item.
[`@has`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L39
[`@matches`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py#L44
[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py
## Testing Locally
Some features of the generated `HTML` documentation might require local

14
src/rustdoc-internals/rustdoc-gui-test-suite.md Normal file
View File

@ -0,0 +1,14 @@
# The `rustdoc-gui` test suite
> **FIXME**: This section is a stub. Please help us flesh it out!
This page is about the test suite named `rustdoc-gui` used to test the "GUI" of `rustdoc` (i.e., the HTML/JS/CSS as rendered in a browser).
For other rustdoc-specific test suites, see [Rustdoc test suites].
These use a NodeJS-based tool called [`browser-UI-test`] that uses [puppeteer] to run tests in a headless browser and check rendering and interactivity. For information on how to write this form of test, see [`tests/rustdoc-gui/README.md`][rustdoc-gui-readme] as well as [the description of the `.goml` format][goml-script]
[Rustdoc test suites]: ../tests/compiletest.md#rustdoc-test-suites
[`browser-UI-test`]: https://github.com/GuillaumeGomez/browser-UI-test/
[puppeteer]: https://pptr.dev/
[rustdoc-gui-readme]: https://github.com/rust-lang/rust/blob/master/tests/rustdoc-gui/README.md
[goml-script]: https://github.com/GuillaumeGomez/browser-UI-test/blob/master/goml-script.md

3
src/rustdoc-internals/rustdoc-json-test-suite.md Normal file
View File

@ -0,0 +1,3 @@
# The `rustdoc-json` test suite
> **FIXME**: This section is a stub. It will be populated by [PR #2422](https://github.com/rust-lang/rustc-dev-guide/pull/2422/).

196
src/rustdoc-internals/rustdoc-test-suite.md
View File

@ -1,112 +1,190 @@
# The `rustdoc` test suite
This page is specifically about the test suite named `rustdoc`.
For other test suites used for testing rustdoc, see [Rustdoc tests](../rustdoc.md#tests).
This page is about the test suite named `rustdoc` used to test the HTML output of `rustdoc`.
For other rustdoc-specific test suites, see [Rustdoc test suites].
The `rustdoc` test suite is specifically used to test the HTML output of rustdoc.
Each test file in this test suite is simply a Rust source file `file.rs` sprinkled with
so-called *directives* located inside normal Rust code comments.
These come in two flavors: *Compiletest* and *HtmlDocCk*.
This is achieved by means of `htmldocck.py`, a custom checker script that leverages [XPath].
To learn more about the former, read [Compiletest directives].
For the latter, continue reading.
[XPath]: https://en.wikipedia.org/wiki/XPath
Internally, [`compiletest`] invokes the supplementary checker script [`htmldocck.py`].
## Directives
Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments.
[Rustdoc test suites]: ../tests/compiletest.md#rustdoc-test-suites
[`compiletest`]: ../tests/compiletest.md
[`htmldocck.py`]: https://github.com/rust-lang/rust/blob/master/src/etc/htmldocck.py
In addition to the directives listed here,
`rustdoc` tests also support most
[compiletest directives](../tests/directives.html).
## HtmlDocCk Directives
All `PATH`s in directives are relative to the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`),
so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid
having to write a long crate name multiple times.
To avoid repetition, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument.
Directives to HtmlDocCk are assertions that place constraints on the generated HTML.
They look similar to those given to `compiletest` in that they take the form of `//@` comments
but ultimately, they are completey distinct and processed by different programs.
All arguments take the form of quoted strings
(both single and double quotes are supported),
[XPath] is used to query parts of the HTML document tree.
**Introductory example**:
```rust,ignore (illustrative)
//@ has file/type.Alias.html
//@ has - '//*[@class="rust item-decl"]//code' 'type Alias = Option<i32>;'
pub type Alias = Option<i32>;
```
Here, we check that documentation generated for crate `file` contains a page for the
public type alias `Alias` where the code block that is found at the top contains the
expected rendering of the item. The `//*[@class="rust item-decl"]//code` is an XPath
expression.
Conventionally, you place these directives directly above the thing they are meant to test.
Technically speaking however, they don't need to be as HtmlDocCk only looks for the directives.
All directives take a `PATH` argument.
To avoid repetition, `-` can be passed to it to re-use the previous `PATH` argument.
Since the path contains the name of the crate, it is conventional to add a
`#![crate_name = "foo"]` attribute to the crate root to shorten the resulting path.
All arguments take the form of shell-style (single or double) quoted strings,
with the exception of `COUNT` and the special `-` form of `PATH`.
Directives are assertions that place constraints on the generated HTML.
All directives (except `files`) can be negated by putting a `!` in front of their name.
All directives (except `files`) can be *negated* by putting a `!` in front of their name.
Before you add negated directives, please read about [their caveats](#caveats).
Similar to shell commands,
directives can extend across multiple lines if their last char is `\`.
In this case, the start of the next line should be `//`, with no `@`.
For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root.
Use the special string `{{channel}}` in XPaths, `PATTERN` arguments and [snapshot files](#snapshot)
if you'd like to refer to the URL `https://doc.rust-lang.org/CHANNEL` where `CHANNEL` refers to the
current release channel (e.g, `stable` or `nightly`).
Listed below are all possible directives:
[XPath]: https://en.wikipedia.org/wiki/XPath
### `has`
Usage 1: `//@ has PATH`
Usage 2: `//@ has PATH XPATH PATTERN`
> Usage 1: `//@ has PATH`
In the first form, `has` checks that a given file exists.
Check that the file given by `PATH` exists.
In the second form, `has` is an alias for `matches`,
except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
> Usage 2: `//@ has PATH XPATH PATTERN`
### `matches`
Checks that the text of each element / attribute / text selected by `XPATH` in the
whitespace-normalized[^1] file given by `PATH` matches the
(also whitespace-normalized) string `PATTERN`.
Usage: `//@ matches PATH XPATH PATTERN`
Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`.
### `matchesraw`
Usage: `//@ matchesraw PATH PATTERN`
Checks that the contents of the file `PATH` matches the regex `PATTERN`.
**Tip**: If you'd like to avoid whitespace normalization and/or if you'd like to match with a regex,
use `matches` instead.
### `hasraw`
Usage: `//@ hasraw PATH PATTERN`
> Usage: `//@ hasraw PATH PATTERN`
Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
Checks that the contents of the whitespace-normalized[^1] file given by `PATH`
matches the (also whitespace-normalized) string `PATTERN`.
**Tip**: If you'd like to avoid whitespace normalization and / or if you'd like to match with a
regex, use `matchesraw` instead.
### `matches`
> Usage: `//@ matches PATH XPATH PATTERN`
Checks that the text of each element / attribute / text selected by `XPATH` in the
file given by `PATH` matches the Python-flavored[^2] regex `PATTERN`.
### `matchesraw`
> Usage: `//@ matchesraw PATH PATTERN`
Checks that the contents of the file given by `PATH` matches the
Python-flavored[^2] regex `PATTERN`.
### `count`
Usage: `//@ count PATH XPATH COUNT`
> Usage: `//@ count PATH XPATH COUNT`
Checks that there are exactly `COUNT` matches for `XPATH` within the file `PATH`.
Checks that there are exactly `COUNT` matches for `XPATH` within the file given by `PATH`.
### `snapshot`
Usage: `//@ snapshot NAME PATH XPATH`
> Usage: `//@ snapshot NAME PATH XPATH`
Creates a snapshot test named NAME.
A snapshot test captures a subtree of the DOM, at the location
determined by the XPath, and compares it to a pre-recorded value
in a file. The file's name is the test's name with the `.rs` extension
replaced with `.NAME.html`, where NAME is the snapshot's name.
Checks that the element / text selected by `XPATH` in the file given by `PATH` matches the
pre-recorded subtree or text (the "snapshot") in file `FILE_STEM.NAME.html` where `FILE_STEM`
is the file stem of the test file.
htmldocck supports the `--bless` option to accept the current subtree
as expected, saving it to the file determined by the snapshot's name.
compiletest's `--bless` flag is forwarded to htmldocck.
Pass the `--bless` option to `compiletest` to accept the current subtree/text as expected.
This will overwrite the aforementioned file (or create it if it doesn't exist). It will
automatically normalize the channel-dependent URL `https://doc.rust-lang.org/CHANNEL` to
the special string `{{channel}}`.
### `has-dir`
Usage: `//@ has-dir PATH`
> Usage: `//@ has-dir PATH`
Checks for the existence of directory `PATH`.
Checks for the existence of the directory given by `PATH`.
### `files`
Usage: `//@ files PATH ENTRIES`
> Usage: `//@ files PATH ENTRIES`
Checks that the directory `PATH` contains exactly `ENTRIES`.
`ENTRIES` is a python list of strings inside a quoted string,
as if it were to be parsed by `eval`.
(note that the list is actually parsed by `shlex.split`,
so it cannot contain arbitrary python expressions).
Checks that the directory given by `PATH` contains exactly `ENTRIES`.
`ENTRIES` is a Python-like list of strings inside a quoted string.
Example: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'`
**Example**: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'`
[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space. The files themselves are also whitespace-normalized.
[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space.
[^2]: They are Unicode aware (flag `UNICODE` is set), match case-sensitively and in single-line mode.
## Compiletest Directives (Brief)
As mentioned in the introduction, you also have access to [compiletest directives].
Most importantly, they allow you to register auxiliary crates and
to pass flags to the `rustdoc` binary under test.
It's *strongly recommended* to read that chapter if you don't know anything about them yet.
Here are some details that are relevant to this test suite specifically:
* While you can use both `//@ compile-flags` and `//@ doc-flags` to pass flags to `rustdoc`,
prefer to user the latter to show intent. The former is meant for `rustc`.
* Add `//@ build-aux-docs` to the test file that has auxiliary crates to not only compile the
auxiliaries with `rustc` but to also document them with `rustdoc`
## Caveats
Testing for the absence of an element or a piece of text is quite fragile and not very future proof.
It's not unusual that the *shape* of the generated HTML document tree changes from time to time.
This includes for example renamings of CSS classes.
Whenever that happens, *positive* checks will either continue to match the intended element /
attribute / text if their XPath expression is general / loose enough and thus test the correct thing
or they won't in which case they would fail forcing the author of the change tolook at them.
Compare that to *negative* checks (e.g., `//@ !has PATH XPATH PATTERN`) which won't fail if their
XPath expression "no longer" matches. The author who changed "the shape" thus won't get notified and
as a result someone else can unintentionally reintroduce `PATTERN` into the generated docs without
the original negative check failing.
**Note**: Please avoid the use of *negated* checks!
**Tip**: If you can't avoid it, please **always** pair it with an analogous positive check in the
immediate vicinity, so people changing "the shape" have a chance to notice and to update the
negated check!
## Limitations
`htmldocck.py` uses the xpath implementation from the standard library.
HtmlDocCk uses the XPath implementation from the Python standard library.
This leads to several limitations:
* All `XPATH` arguments must start with `//` due to a flaw in the implementation.
* Many XPath features (functions, axies, etc.) are not supported.
* Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags).
Furthmore, compiletest [revisions] are not supported.
[revisions]: ../tests/compiletest.md#revisions
[compiletest directives]: ../tests/directives.md

42
src/rustdoc.md
View File

@ -67,43 +67,29 @@ does is call the `main()` that's in this crate's `lib.rs`, though.)
## Code structure
* All paths in this section are relative to `src/librustdoc` in the rust-lang/rust repository.
All paths in this section are relative to `src/librustdoc/` in the rust-lang/rust repository.
* Most of the HTML printing code is in `html/format.rs` and `html/render/mod.rs`.
It's in a bunch of `fmt::Display` implementations and supplementary
functions.
* The types that got `Display` impls above are defined in `clean/mod.rs`, right
next to the custom `Clean` trait used to process them out of the rustc HIR.
It's in a bunch of functions returning `impl std::fmt::Display`.
* The data types that get rendered by the functions mentioned above are defined in `clean/types.rs`.
The functions responsible for creating them from the `HIR` and the `rustc_middle::ty` IR
live in `clean/mod.rs`.
* The bits specific to using rustdoc as a test harness are in
`doctest.rs`.
* The Markdown renderer is loaded up in `html/markdown.rs`, including functions
for extracting doctests from a given block of Markdown.
* Frontend CSS and JavaScript are stored in `html/static/`.
* Re. JavaScript, type annotations are written using [TypeScript-flavored JSDoc]
comments and an external `.d.ts` file.
This way, the code itself remains plain, valid JavaScript.
We only use `tsc` as a linter.
[TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html
## Tests
* Tests on search engine and index are located in `tests/rustdoc-js` and `tests/rustdoc-js-std`.
The format is specified
[in the search guide](rustdoc-internals/search.md#testing-the-search-engine).
* Tests on the "UI" of rustdoc (the terminal output it produces when run) are in
`tests/rustdoc-ui`
* Tests on the "GUI" of rustdoc (the HTML, JS, and CSS as rendered in a browser)
are in `tests/rustdoc-gui`. These use a [NodeJS tool called
browser-UI-test](https://github.com/GuillaumeGomez/browser-UI-test/) that uses
puppeteer to run tests in a headless browser and check rendering and
interactivity. For information on how to write this form of test,
see [`tests/rustdoc-gui/README.md`][rustdoc-gui-readme]
as well as [the description of the `.goml` format][goml-script]
* Tests on the structure of rustdoc HTML output are located in `tests/rustdoc`,
where they're handled by the test runner of bootstrap and
the supplementary script `src/etc/htmldocck.py`.
[These tests have several extra directives available to them](./rustdoc-internals/rustdoc-test-suite.md).
* Additionally, JavaScript type annotations are written using [TypeScript-flavored JSDoc]
comments and an external d.ts file. The code itself is plain, valid JavaScript; we only
use tsc as a linter.
[TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html
[rustdoc-gui-readme]: https://github.com/rust-lang/rust/blob/master/tests/rustdoc-gui/README.md
[goml-script]: https://github.com/GuillaumeGomez/browser-UI-test/blob/master/goml-script.md
`rustdoc`'s integration tests are split across several test suites.
See [Rustdoc tests suites](tests/compiletest.md#rustdoc-test-suites) for more details.
## Constraints

30
src/tests/compiletest.md
View File

@ -56,6 +56,9 @@ incremental compilation. The various suites are defined in
The following test suites are available, with links for more information:
[`tests`]: https://github.com/rust-lang/rust/blob/master/tests
[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
### Compiler-specific test suites
| Test suite | Purpose |
@ -71,6 +74,7 @@ The following test suites are available, with links for more information:
| [`mir-opt`](#mir-opt-tests) | Check MIR generation and optimizations |
| [`coverage`](#coverage-tests) | Check coverage instrumentation |
| [`coverage-run-rustdoc`](#coverage-tests) | `coverage` tests that also run instrumented doctests |
| [`crashes`](#crashes-tests) | Check that the compiler ICEs/panics/crashes on certain inputs to catch accidental fixes |
### General purpose test suite
@ -78,19 +82,23 @@ The following test suites are available, with links for more information:
### Rustdoc test suites
See [Rustdoc tests](../rustdoc.md#tests) for more details.
| Test suite | Purpose |
|--------------------------------------|--------------------------------------------------------------------------|
| [`rustdoc`][rustdoc-html-tests] | Check HTML output of `rustdoc` |
| [`rustdoc-gui`][rustdoc-gui-tests] | Check `rustdoc`'s GUI using a web browser |
| [`rustdoc-js`][rustdoc-js-tests] | Check `rustdoc`'s search engine and index |
| [`rustdoc-js-std`][rustdoc-js-tests] | Check `rustdoc`'s search engine and index on the std library docs |
| [`rustdoc-json`][rustdoc-json-tests] | Check JSON output of `rustdoc` |
| `rustdoc-ui` | Check terminal output of `rustdoc` ([see also](ui.md)) |
| Test suite | Purpose |
|------------------|--------------------------------------------------------------------------|
| `rustdoc` | Check `rustdoc` generated files contain the expected documentation |
| `rustdoc-gui` | Check `rustdoc`'s GUI using a web browser |
| `rustdoc-js` | Check `rustdoc` search is working as expected |
| `rustdoc-js-std` | Check rustdoc search is working as expected specifically on the std docs |
| `rustdoc-json` | Check JSON output of `rustdoc` |
| `rustdoc-ui` | Check terminal output of `rustdoc` |
Some rustdoc-specific tests can also be found in `ui/rustdoc/`.
These check rustdoc-related or -specific lints that (also) run as part of `rustc`, not (only) `rustdoc`.
Run-make tests pertaining to rustdoc are typically named `run-make/rustdoc-*/`.
[`tests`]: https://github.com/rust-lang/rust/blob/master/tests
[`src/tools/compiletest/src/common.rs`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest/src/common.rs
[rustdoc-html-tests]: ../rustdoc-internals/rustdoc-test-suite.md
[rustdoc-gui-tests]: ../rustdoc-internals/rustdoc-gui-test-suite.md
[rustdoc-js-tests]: ../rustdoc-internals/search.md#testing-the-search-engine
[rustdoc-json-tests]: ../rustdoc-internals/rustdoc-json-test-suite.md
### Pretty-printer tests

13
src/tests/directives.md
View File

@ -261,7 +261,7 @@ Consider writing the test as a proper incremental test instead.
| Directive | Explanation | Supported test suites | Possible values |
|-------------|--------------------------------------------------------------|------------------------------------------|---------------------------|
| `doc-flags` | Flags passed to `rustdoc` when building the test or aux file | `rustdoc`, `rustdoc-js`, `rustdoc-json` | Any valid `rustdoc` flags |
| `doc-flags` | Flags passed to `rustdoc` when building the test or aux file | `rustdoc`, `rustdoc-js`, `rustdoc-json` | Any valid `rustdoc` flags |
<!--
**FIXME(rustdoc)**: what does `check-test-line-numbers-match` do?
@ -269,6 +269,17 @@ Asked in
<https://rust-lang.zulipchat.com/#narrow/stream/266220-t-rustdoc/topic/What.20is.20the.20.60check-test-line-numbers-match.60.20directive.3F>.
-->
#### Test-suite-specific directives
The test suites [`rustdoc`][rustdoc-html-tests], [`rustdoc-js`/`rustdoc-js-std`][rustdoc-js-tests]
and [`rustdoc-json`][rustdoc-json-tests] each feature an additional set of directives whose basic
syntax resembles the one of compiletest directives but which are ultimately read and checked by
separate tools. For more information, please read their respective chapters as linked above.
[rustdoc-html-tests]: ../rustdoc-internals/rustdoc-test-suite.md
[rustdoc-js-tests]: ../rustdoc-internals/search.html#testing-the-search-engine
[rustdoc-json-tests]: ../rustdoc-internals/rustdoc-json-test-suite.md
### Pretty printing
See [Pretty-printer](compiletest.md#pretty-printer-tests).