Merge pull request #2295 from lolbinarycat/rustdoc-htmldocck

add new section on the `rustdoc` test suite

src/SUMMARY.md

@@ -95,7 +95,7 @@
 - [Parallel Compilation](./parallel-rustc.md)
 - [Rustdoc internals](./rustdoc-internals.md)
     - [Search](./rustdoc-internals/search.md)
-
+	- [The `rustdoc` test suite](./rustdoc-internals/rustdoc-test-suite.md)
 # Source Code Representation
 
 - [Prologue](./part-3-intro.md)

src/rustdoc-internals/rustdoc-test-suite.md

@@ -0,0 +1,111 @@
+# The `rustdoc` test suite
+
+This page is specifically about the test suite named `rustdoc`, for other test suites used for testing rustdoc, see [Rustdoc tests](../rustdoc.md#tests).
+
+The `rustdoc` test suite is specifically used to test the HTML output of rustdoc.
+
+This is achived by means of `htmldocck.py`, a custom checker script that leverages [XPath].
+
+[XPath]: https://en.wikipedia.org/wiki/XPath
+
+## Directives
+Directives to htmldocck are similar to those given to `compiletest` in that they take the form of `//@` comments.
+
+In addition to the directives listed here,
+`rustdoc` tests also support most
+[compiletest directives](../tests/directives.html).
+
+All `PATH`s in directives are relative to the the rustdoc output directory (`build/TARGET/test/rustdoc/TESTNAME`),
+so it is conventional to use a `#![crate_name = "foo"]` attribute to avoid
+having to write a long crate name multiple times.
+To avoid repetion, `-` can be used in any `PATH` argument to re-use the previous `PATH` argument.
+
+All arguments take the form of quoted strings
+(both single and double quotes are supported),
+with the exception of `COUNT` and the special `-` form of `PATH`.
+
+Directives are assertions that place constraints on the generated HTML.
+
+All directives (except `files`) can be negated by putting a `!` in front of their name.
+
+Similar to shell commands,
+directives can extend across multiple lines if their last char is `\`.
+In this case, the start of the next line should be `//`, with no `@`.
+
+For example, `//@ !has 'foo/struct.Bar.html'` checks that crate `foo` does not have a page for a struct named `Bar` in the crate root.
+
+### `has`
+
+Usage 1: `//@ has PATH`
+Usage 2: `//@ has PATH XPATH PATTERN`
+
+In the first form, `has` checks that a given file exists.
+
+In the second form, `has` is an alias for `matches`,
+except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
+
+### `matches`
+
+Usage: `//@ matches PATH XPATH PATTERN`
+
+Checks that the text of each element selected by `XPATH` in `PATH` matches the python-flavored regex `PATTERN`.
+
+### `matchesraw`
+
+Usage: `//@ matchesraw PATH PATTERN`
+
+Checks that the contents of the file `PATH` matches the regex `PATTERN`.
+
+### `hasraw`
+
+Usage: `//@ hasraw PATH PATTERN`
+
+Same as `matchesraw`, except `PATTERN` is a whitespace-normalized[^1] string instead of a regex.
+
+### `count`
+
+Usage: `//@ count PATH XPATH COUNT`
+
+Checks that there are exactly `COUNT` matches for `XPATH` within the file `PATH`.
+
+### `snapshot`
+
+Usage: `//@ snapshot NAME PATH XPATH`
+
+Creates a snapshot test named NAME.
+A snapshot test captures a subtree of the DOM, at the location
+determined by the XPath, and compares it to a pre-recorded value
+in a file. The file's name is the test's name with the `.rs` extension
+replaced with `.NAME.html`, where NAME is the snapshot's name.
+
+htmldocck supports the `--bless` option to accept the current subtree
+as expected, saving it to the file determined by the snapshot's name.
+compiletest's `--bless` flag is forwarded to htmldocck.
+
+### `has-dir`
+
+Usage: `//@ has-dir PATH`
+
+Checks for the existance of directory `PATH`.
+
+### `files`
+
+Usage: `//@ files PATH ENTRIES`
+
+Checks that the directory `PATH` contains exactly `ENTRIES`.
+`ENTRIES` is a python list of strings inside a quoted string,
+as if it were to be parsed by `eval`.
+(note that the list is actually parsed by `shlex.split`,
+so it cannot contain arbitrary python expressions).
+
+Example: `//@ files "foo/bar" '["index.html", "sidebar-items.js"]'`
+
+[^1]: Whitespace normalization means that all spans of consecutive whitespace are replaced with a single space.  The files themselves are also whitespace-normalized.
+
+## Limitations
+`htmldocck.py` uses the xpath implementation from the standard library.
+This leads to several limitations:
+* All `XPATH` arguments must start with `//` due to a flaw in the implemention.
+* Many XPath features (functions, axies, etc.) are not supported.
+* Only well-formed HTML can be parsed (hopefully rustdoc doesn't output mismatched tags).
+

src/rustdoc.md

@@ -77,27 +77,27 @@ does is call the `main()` that's in this crate's `lib.rs`, though.)
   `doctest.rs`.
 * The Markdown renderer is loaded up in `html/markdown.rs`, including functions
   for extracting doctests from a given block of Markdown.
-* The tests on the structure of rustdoc HTML output are located in `tests/rustdoc`, where
-  they're handled by the test runner of bootstrap and the supplementary script
-  `src/etc/htmldocck.py`.
 * Frontend CSS and JavaScript are stored in `html/static/`.
 
 ## Tests
 
-* All paths in this section are relative to `tests` in the rust-lang/rust repository.
-* Tests on search engine and index are located in `rustdoc-js` and `rustdoc-js-std`.
+* Tests on search engine and index are located in `tests/rustdoc-js` and `tests/rustdoc-js-std`.
   The format is specified
   [in the search guide](rustdoc-internals/search.md#testing-the-search-engine).
 * Tests on the "UI" of rustdoc (the terminal output it produces when run) are in
-  `rustdoc-ui`
+  `tests/rustdoc-ui`
 * Tests on the "GUI" of rustdoc (the HTML, JS, and CSS as rendered in a browser)
-  are in `rustdoc-gui`. These use a [NodeJS tool called
+  are in `tests/rustdoc-gui`. These use a [NodeJS tool called
   browser-UI-test](https://github.com/GuillaumeGomez/browser-UI-test/) that uses
   puppeteer to run tests in a headless browser and check rendering and
   interactivity.
 * Additionally, JavaScript type annotations are written using [TypeScript-flavored JSDoc]
   comments and an external d.ts file. The code itself is plain, valid JavaScript; we only
   use tsc as a linter.
+* The tests on the structure of rustdoc HTML output are located in `tests/rustdoc`,
+  where they're handled by the test runner of bootstrap and
+  the supplementary script `src/etc/htmldocck.py`.
+  [These tests have several extra directives available to them](./rustdoc-internals/rustdoc-test-suite.md).
 
 [TypeScript-flavored JSDoc]: https://www.typescriptlang.org/docs/handbook/jsdoc-supported-types.html