mirror
/
rustc-dev-guide
mirror of https://github.com/rust-lang/rustc-dev-guide.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

expand notes on expansion heirarchies

This commit is contained in:
mark 2020-04-30 20:08:36 -05:00 committed by Who? Me?!
parent ba8620f34a
commit f05ff9c30d
1 changed files with 128 additions and 57 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

185
src/macro-expansion.md
View File

@ -163,82 +163,153 @@ only within the macro (i.e. it should not be visible outside the macro).
[code_parse_int]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_expand/mbe/macro_parser/fn.parse_tt.html
[parsing]: ./the-parser.html
TODO: expand these notes
The context is attached to AST nodes. All AST nodes generated by macros have
context attached. Additionally, there may be other nodes that have context
attached, such as some desugared syntax (non-macro-expanded nodes are
considered to just have the "root" context, as described below).
- Many AST nodes have some sort of syntax context, especially nodes from macros.
- When we ask what is the syntax context of a node, the answer actually differs by what we are trying to do. Thus, we don't just keep track of a single context. There are in fact 3 different types of context used for different things.
- Each type of context is tracked by an "expansion heirarchy". As we expand macros, new macro calls or macro definitions may be generated, leading to some nesting. This nesting is where the heirarchies come from. Each heirarchy tracks some different aspect, though, as we will see.
- There are 3 expansion heirarchies
- All macros receive an integer ID assigned continuously starting from 0 as we discover new macro calls
- This is used as the `expn_id` where needed.
- All heirarchies start at ExpnId::root, which is its own parent
- The context of a node consists of a chain of expansions leading to `ExpnId::root`. A non-macro-expanded node has syntax context 0 (`SyntaxContext::empty()`) which represents just the root node.
- There are vectors in `HygieneData` that contain expansion info.
- There are entries here for both `SyntaxContext::empty()` and `ExpnId::root`, but they aren't used much.
Because macros invocations and definitions can be nested, the syntax context of
a node must be a heirarchy. For example, if we expand a macro and there is
another macro invocation or definition in the generated output, then the syntax
context should reflex the nesting.
1. Tracks expansion order: when a macro invocation is in the output of another macro.
...
expn_id2
expn_id1
InternalExpnData::parent is the child->parent link. That is the expn_id1 points to expn_id2 points to ...
However, it turns out that there are actually a few types of context we may
want to track for different purposes. Thus, there not just one but _three_
expansion heirarchies that together comprise the hygiene information for a
crate.
Ex:
macro_rules! foo { () => { println!(); } }
fn main() { foo!(); }
All of these heirarchies need some sort of "macro ID" to identify individual
elements in the chain of expansions. This ID is [`ExpnId`]. All macros receive
an integer ID, assigned continuously starting from 0 as we discover new macro
calls. All heirarchies start at [`ExpnId::root()`][rootid], which is its own
parent.
// Then AST nodes that are finally generated would have parent(expn_id_println) -> parent(expn_id_foo), right?
The actual heirarchies are stored in [`HygieneData`][hd], and all of the
hygiene-related algorithms are implemented in [`rustc_span::hygiene`][hy], with
the exception of some hacks [`Resolver::resolve_crate_root`][hacks].
2. Tracks macro definitions: when we are expanding one macro another macro definition is revealed in its output.
...
SyntaxContext2
SyntaxContext1
SyntaxContextData::parent is the child->parent link here.
SyntaxContext is the whole chain in this hierarchy, and SyntaxContextData::outer_expns are individual elements in the chain.
[`ExpnId`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnId.html
[rootid]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnId.html#method.root
[hd]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.HygieneData.html
[hy]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/index.html
[hacks]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_resolve/struct.Resolver.html#method.resolve_crate_root
- For built-in macros (e.g. `line!()`) or stable proc macros: tokens produced by the macro are given the context `SyntaxContext::empty().apply_mark(expn_id)`
- Such macros are considered to have been defined at the root.
- For proc macros this is because they are always cross-crate and we don't have cross-crate hygiene implemented.
### The Expansion Order Heirarchy
The second hierarchy has the context transplantation hack. See https://github.com/rust-lang/rust/pull/51762#issuecomment-401400732.
The first heirarchy tracks the order of expansions, i.e., when a macro
invocation is in the output of another macro.
If the token had context X before being produced by a macro then after being produced by the macro it has context X -> macro_id.
Here, the children in the heirarchy will be the "innermost" tokens.
[`ExpnData::parent`][edp] tracks the child -> parent link in this heirarchy.
Ex:
```rust
macro m() { ident }
```
[edp]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnData.html#structfield.parent
Here `ident` originally has context SyntaxContext::root(). `ident` has context ROOT -> id(m) after it's produced by m.
The "chaining operator" is `apply_mark` in compiler code.
For example,
Ex:
```rust,ignore
macro_rules! foo { () => { println!(); } }
```rust
macro m() { macro n() { ident } }
```
In this example the ident has context ROOT originally, then ROOT -> id(m), then ROOT -> id(m) -> id(n).
fn main() { foo!(); }
```
Note that these chains are not entirely determined by their last element, in other words ExpnId is not isomorphic to SyntaxCtxt.
In this code, the AST nodes that are finally generated would have heirarchy:
Ex:
```rust
macro m($i: ident) { macro n() { ($i, bar) } }
```
root
expn_id_foo
expn_id_println
```
m!(foo);
```
### The Macro Definition Heirarchy
After all expansions, foo has context ROOT -> id(n) and bar has context ROOT -> id(m) -> id(n)
The second heirarchy tracks the order of macro definitions, i.e., when we are
expanding one macro another macro definition is revealed in its output. This
one is a bit tricky and more complex than the other two heirarchies.
3. Call-site: tracks the location of the macro invocation.
Ex:
If foo!(bar!(ident)) expands into ident
then hierarchy 1 is root -> foo -> bar -> ident
but hierarchy 3 is root -> ident
Here, [`SyntaxContextData::parent`][scdp] is the child -> parent link here.
[`SyntaxContext`][sc] is the whole chain in this hierarchy, and
[`SyntaxContextData::outer_expns`][scdoe] are individual elements in the chain.
The "chaining operator" is [`SyntaxContext::apply_mark`][am] in compiler code.
ExpnInfo::call_site is the child-parent link in this case.
[scdp]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContextData.html#structfield.parent
[sc]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html
[scdoe]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContextData.html#structfield.outer_expn
[am]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html#method.apply_mark
- Hygiene-related algorithms are entirely in hygiene.rs
- Some hacks in `resolve_crate_root`, though.
For built-in macros, we use the context:
`SyntaxContext::empty().apply_mark(expn_id)`, and such macros are considered to
be defined at the heirarchy root. We do the same for proc-macros because we
haven't implemented cross-crate hygiene yet.
If the token had context `X` before being produced by a macro then after being
produced by the macro it has context `X -> macro_id`. Here are some examples:
Example 0:
```rust,ignore
macro m() { ident }
m!();
```
Here `ident` originally has context [`SyntaxContext::root()`][scr]. `ident` has
context `ROOT -> id(m)` after it's produced by `m`.
[scr]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.SyntaxContext.html#method.root
Example 1:
```rust,ignore
macro m() { macro n() { ident } }
m!();
n!();
```
In this example the `ident` has context `ROOT` originally, then `ROOT -> id(m)`
after the first expansion, then `ROOT -> id(m) -> id(n)`.
Example 2:
Note that these chains are not entirely determined by their last element, in
other words `ExpnId` is not isomorphic to `SyntaxContext`.
```rust,ignore
macro m($i: ident) { macro n() { ($i, bar) } }
m!(foo);
```
After all expansions, `foo` has context `ROOT -> id(n)` and `bar` has context
`ROOT -> id(m) -> id(n)`.
Finally, one last thing to mention is that currently, this heirarchy is subject
to the ["context transplantation hack"][hack]. Basically, the more modern (and
experimental) `macro` macros have stronger hygiene than the older MBE system,
but this can result in weird interactions between the two. The hack is intended
to make things "just work" for now.
[hack]: https://github.com/rust-lang/rust/pull/51762#issuecomment-401400732
### The Call-site Heirarchy
The third and final heirarchy tracks the location of macro invocations.
In this heirarchy [`ExpnData::call_site`][callsite] is the child -> parent link.
[callsite]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/hygiene/struct.ExpnData.html#structfield.call_site
Here is an example:
```rust,ignore
macro bar($i: ident) { $i }
macro foo($i: ident) { $i }
foo!(bar!(baz));
```
For the `baz` AST node in the final output, the first heirarchy is `ROOT ->
id(foo) -> id(bar) -> baz`, while the third heirarchy is `ROOT -> baz`.
## Producing Macro Output