add discussion transcript so we don't lose it
This commit is contained in:
Mark Mansi
Who? Me?!
parent
2dabf0f58d
commit
a1d1860a11
|
|
@ -210,3 +210,706 @@ TODO: maybe something about macros 2.0?
|
|||
[code_mr]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax_expand/mbe/macro_rules
|
||||
[code_parse_int]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax_expand/mbe/macro_parser/fn.parse.html
|
||||
[parsing]: ./the-parser.html
|
||||
|
||||
|
||||
# Discussion about hygiene
|
||||
|
||||
The rest of this chapter is a dump of a discussion between `mark-i-m` and
|
||||
`petrochenkov` about Macro Expansion and Hygiene. I am pasting it here so that
|
||||
it never gets lost until we can make it into a proper chapter.
|
||||
|
||||
```txt
|
||||
mark-i-m: @Vadim Petrochenkov Hi :wave:
|
||||
I was wondering if you would have a chance sometime in the next month or so to
|
||||
just have a zulip discussion where you tell us (WG-learning) everything you
|
||||
know about macros/expansion/hygiene. We were thinking this could be less formal
|
||||
(and less work for you) than compiler lecture series lecture... thoughts?
|
||||
|
||||
mark-i-m: The goal is to fill out that long-standing gap in the rustc-guide
|
||||
|
||||
Vadim Petrochenkov: Ok, I'm at UTC+03:00 and generally available in the
|
||||
evenings (or weekends).
|
||||
|
||||
mark-i-m: @Vadim Petrochenkov Either of those works for me (your evenings are
|
||||
about lunch time for me :) ) Is there a particular date that would work best
|
||||
for you?
|
||||
|
||||
mark-i-m: @WG-learning Does anyone else have a preferred date?
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
Is there a particular date that would work best for you?
|
||||
|
||||
Nah, not much difference. (If something changes for a specific day, I'll
|
||||
notify.)
|
||||
|
||||
Santiago Pastorino: week days are better, but I'd say let's wait for @Vadim
|
||||
Petrochenkov to say when they are ready for it and we can set a date
|
||||
|
||||
Santiago Pastorino: also, we should record this so ... I guess it doesn't
|
||||
matter that much when :)
|
||||
|
||||
mark-i-m:
|
||||
|
||||
also, we should record this so ... I guess it doesn't matter that much when
|
||||
:)
|
||||
|
||||
@Santiago Pastorino My thinking was to just use zulip, so we would have the log
|
||||
|
||||
mark-i-m: @Vadim Petrochenkov @WG-learning How about 2 weeks from now: July 24
|
||||
at 5pm UTC time (if I did the math right, that should be evening for Vadim)
|
||||
|
||||
Amanjeev Sethi: i can try and do this but I am starting a new job that week so
|
||||
cannot promise.
|
||||
|
||||
Santiago Pastorino:
|
||||
|
||||
Vadim Petrochenkov @WG-learning How about 2 weeks from now: July 24 at 5pm
|
||||
UTC time (if I did the math right, that should be evening for Vadim)
|
||||
|
||||
works perfect for me
|
||||
|
||||
Santiago Pastorino: @mark-i-m I have access to the compiler calendar so I can
|
||||
add something there
|
||||
|
||||
Santiago Pastorino: let me know if you want to add an event to the calendar, I
|
||||
can do that
|
||||
|
||||
Santiago Pastorino: how long it would be?
|
||||
|
||||
mark-i-m:
|
||||
|
||||
let me know if you want to add an event to the calendar, I can do that
|
||||
|
||||
mark-i-m: That could be good :+1:
|
||||
|
||||
mark-i-m:
|
||||
|
||||
how long it would be?
|
||||
|
||||
Let's start with 30 minutes, and if we need to schedule another we cna
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
5pm UTC
|
||||
|
||||
1-2 hours later would be better, 5pm UTC is not evening enough.
|
||||
|
||||
Vadim Petrochenkov: How exactly do you plan the meeting to go (aka how much do
|
||||
I need to prepare)?
|
||||
|
||||
Santiago Pastorino:
|
||||
|
||||
5pm UTC
|
||||
|
||||
1-2 hours later would be better, 5pm UTC is not evening enough.
|
||||
|
||||
Scheduled for 7pm UTC then
|
||||
|
||||
Santiago Pastorino:
|
||||
|
||||
How exactly do you plan the meeting to go (aka how much do I need to
|
||||
prepare)?
|
||||
|
||||
/cc @mark-i-m
|
||||
|
||||
mark-i-m: @Vadim Petrochenkov
|
||||
|
||||
How exactly do you plan the meeting to go (aka how much do I need to
|
||||
prepare)?
|
||||
|
||||
My hope was that this could be less formal than for a compiler lecture series,
|
||||
but it would be nice if you could have in your mind a tour of the design and
|
||||
the code
|
||||
|
||||
That is, imagine that a new person was joining the compiler team and needed to
|
||||
get up to speed about macros/expansion/hygiene. What would you tell such a
|
||||
person?
|
||||
|
||||
mark-i-m: @Vadim Petrochenkov Are we still on for tomorrow at 7pm UTC?
|
||||
|
||||
Vadim Petrochenkov: Yes.
|
||||
|
||||
Santiago Pastorino: @Vadim Petrochenkov @mark-i-m I've added an event on rust
|
||||
compiler team calendar
|
||||
|
||||
mark-i-m: @WG-learning @Vadim Petrochenkov Hello!
|
||||
|
||||
mark-i-m: We will be starting in ~7 minutes
|
||||
|
||||
mark-i-m: :wave:
|
||||
|
||||
Vadim Petrochenkov: I'm here.
|
||||
|
||||
mark-i-m: Cool :)
|
||||
|
||||
Santiago Pastorino: hello @Vadim Petrochenkov
|
||||
|
||||
mark-i-m: Shall we start?
|
||||
|
||||
mark-i-m: First off, @Vadim Petrochenkov Thanks for doing this!
|
||||
|
||||
Vadim Petrochenkov: Here's some preliminary data I prepared.
|
||||
|
||||
Vadim Petrochenkov: Below I'll assume #62771 and #62086 has landed.
|
||||
|
||||
Vadim Petrochenkov: Where to find the code: libsyntax_pos/hygiene.rs -
|
||||
structures related to hygiene and expansion that are kept in global data (can
|
||||
be accessed from any Ident without any context) libsyntax_pos/lib.rs - some
|
||||
secondary methods like macro backtrace using primary methods from hygiene.rs
|
||||
libsyntax_ext - implementations of built-in macros (including macro attributes
|
||||
and derives) and some other early code generation facilities like injection of
|
||||
standard library imports or generation of test harness. libsyntax/config.rs -
|
||||
implementation of cfg/cfg_attr (they treated specially from other macros),
|
||||
should probably be moved into libsyntax/ext. libsyntax/tokenstream.rs +
|
||||
libsyntax/parse/token.rs - structures for compiler-side tokens, token trees,
|
||||
and token streams. libsyntax/ext - various expansion-related stuff
|
||||
libsyntax/ext/base.rs - basic structures used by expansion
|
||||
libsyntax/ext/expand.rs - some expansion structures and the bulk of expansion
|
||||
infrastructure code - collecting macro invocations, calling into resolve for
|
||||
them, calling their expanding functions, and integrating the results back into
|
||||
AST libsyntax/ext/placeholder.rs - the part of expand.rs responsible for
|
||||
"integrating the results back into AST" basicallly, "placeholder" is a
|
||||
temporary AST node replaced with macro expansion result nodes
|
||||
libsyntax/ext/builer.rs - helper functions for building AST for built-in macros
|
||||
in libsyntax_ext (and user-defined syntactic plugins previously), can probably
|
||||
be moved into libsyntax_ext these days libsyntax/ext/proc_macro.rs +
|
||||
libsyntax/ext/proc_macro_server.rs - interfaces between the compiler and the
|
||||
stable proc_macro library, converting tokens and token streams between the two
|
||||
representations and sending them through C ABI libsyntax/ext/tt -
|
||||
implementation of macro_rules, turns macro_rules DSL into something with
|
||||
signature Fn(TokenStream) -> TokenStream that can eat and produce tokens,
|
||||
@mark-i-m knows more about this librustc_resolve/macros.rs - resolving macro
|
||||
paths, validating those resolutions, reporting various "not found"/"found, but
|
||||
it's unstable"/"expected x, found y" errors librustc/hir/map/def_collector.rs +
|
||||
librustc_resolve/build_reduced_graph.rs - integrate an AST fragment freshly
|
||||
expanded from a macro into various parent/child structures like module
|
||||
hierarchy or "definition paths"
|
||||
|
||||
Primary structures: HygieneData - global piece of data containing hygiene and
|
||||
expansion info that can be accessed from any Ident without any context ExpnId -
|
||||
ID of a macro call or desugaring (and also expansion of that call/desugaring,
|
||||
depending on context) ExpnInfo/InternalExpnData - a subset of properties from
|
||||
both macro definition and macro call available through global data
|
||||
SyntaxContext - ID of a chain of nested macro definitions (identified by
|
||||
ExpnIds) SyntaxContextData - data associated with the given SyntaxContext,
|
||||
mostly a cache for results of filtering that chain in different ways Span - a
|
||||
code location + SyntaxContext Ident - interned string (Symbol) + Span, i.e. a
|
||||
string with attached hygiene data TokenStream - a collection of TokenTrees
|
||||
TokenTree - a token (punctuation, identifier, or literal) or a delimited group
|
||||
(anything inside ()/[]/{}) SyntaxExtension - a lowered macro representation,
|
||||
contains its expander function transforming a tokenstream or AST into
|
||||
tokenstream or AST + some additional data like stability, or a list of unstable
|
||||
features allowed inside the macro. SyntaxExtensionKind - expander functions
|
||||
may have several different signatures (take one token stream, or two, or a
|
||||
piece of AST, etc), this is an enum that lists them
|
||||
ProcMacro/TTMacroExpander/AttrProcMacro/MultiItemModifier - traits representing
|
||||
the expander signatures (TODO: change and rename the signatures into something
|
||||
more consistent) trait Resolver - a trait used to break crate dependencies (so
|
||||
resolver services can be used in libsyntax, despite librustc_resolve and pretty
|
||||
much everything else depending on libsyntax) ExtCtxt/ExpansionData - various
|
||||
intermediate data kept and used by expansion infra in the process of its work
|
||||
AstFragment - a piece of AST that can be produced by a macro (may include
|
||||
multiple homogeneous AST nodes, like e.g. a list of items) Annotatable - a
|
||||
piece of AST that can be an attribute target, almost same thing as AstFragment
|
||||
except for types and patterns that can be produced by macros but cannot be
|
||||
annotated with attributes (TODO: Merge into AstFragment) trait MacResult - a
|
||||
"polymorphic" AST fragment, something that can turn into a different
|
||||
AstFragment depending on its context (aka AstFragmentKind - item, or
|
||||
expression, or pattern etc.) Invocation/InvocationKind - a structure describing
|
||||
a macro call, these structures are collected by the expansion infra
|
||||
(InvocationCollector), queued, resolved, expanded when resolved, etc.
|
||||
|
||||
Primary algorithms / actions: TODO
|
||||
|
||||
mark-i-m: Very useful :+1:
|
||||
|
||||
mark-i-m: @Vadim Petrochenkov Zulip doesn't have an indication of typing, so
|
||||
I'm not sure if you are waiting for me or not
|
||||
|
||||
Vadim Petrochenkov: The TODO part should be about how a crate transitions from
|
||||
the state "macros exist as written in source" to "all macros are expanded", but
|
||||
I didn't write it yet.
|
||||
|
||||
Vadim Petrochenkov: (That should probably better happen off-line.)
|
||||
|
||||
Vadim Petrochenkov: Now, if you have any questions?
|
||||
|
||||
mark-i-m: Thanks :)
|
||||
|
||||
mark-i-m: /me is still reading :P
|
||||
|
||||
mark-i-m: Ok
|
||||
|
||||
mark-i-m: So I guess my first question is about hygiene, since that remains the
|
||||
most mysterious to me... My understanding is that the parser outputs AST nodes,
|
||||
where each node has a Span
|
||||
|
||||
mark-i-m: In the absence of macros and desugaring, what does the syntax context
|
||||
of an AST node look like?
|
||||
|
||||
mark-i-m: @Vadim Petrochenkov
|
||||
|
||||
Vadim Petrochenkov: Not each node, but many of them. When a node is not
|
||||
macro-expanded, its context is 0.
|
||||
|
||||
Vadim Petrochenkov: aka SyntaxContext::empty()
|
||||
|
||||
Vadim Petrochenkov: it's a chain that consists of one expansion - expansion 0
|
||||
aka ExpnId::root.
|
||||
|
||||
mark-i-m: Do all expansions start at root?
|
||||
|
||||
Vadim Petrochenkov: Also, SyntaxContext:empty() is its own father.
|
||||
|
||||
mark-i-m: Is this actually stored somewhere or is it a logical value?
|
||||
|
||||
Vadim Petrochenkov: All expansion hyerarchies (there are several of them) start
|
||||
at ExpnId::root.
|
||||
|
||||
Vadim Petrochenkov: Vectors in HygieneData has entries for both ctxt == 0 and
|
||||
expn_id == 0.
|
||||
|
||||
Vadim Petrochenkov: I don't think anyone looks into them much though.
|
||||
|
||||
mark-i-m: Ok
|
||||
|
||||
Vadim Petrochenkov: Speaking of multiple hierarchies...
|
||||
|
||||
mark-i-m: Go ahead :)
|
||||
|
||||
Vadim Petrochenkov: One is parent (expn_id1) -> parent(expn_id2) -> ...
|
||||
|
||||
Vadim Petrochenkov: This is the order in which macros are expanded.
|
||||
|
||||
Vadim Petrochenkov: Well.
|
||||
|
||||
Vadim Petrochenkov: When we are expanding one macro another macro is revealed
|
||||
in its output.
|
||||
|
||||
Vadim Petrochenkov: That's the parent-child relation in this hierarchy.
|
||||
|
||||
Vadim Petrochenkov: InternalExpnData::parent is the child->parent link.
|
||||
|
||||
mark-i-m: So in the above chain expn_id1 is the child?
|
||||
|
||||
Vadim Petrochenkov: Yes.
|
||||
|
||||
Vadim Petrochenkov: The second one is parent (SyntaxContext1) ->
|
||||
parent(SyntaxContext2) -> ...
|
||||
|
||||
Vadim Petrochenkov: This is about nested macro definitions. When we are
|
||||
expanding one macro another macro definition is revealed in its output.
|
||||
|
||||
Vadim Petrochenkov: SyntaxContextData::parent is the child->parent link here.
|
||||
|
||||
Vadim Petrochenkov: So, SyntaxContext is the whole chain in this hierarchy, and
|
||||
outer_expns are individual elements in the chain.
|
||||
|
||||
mark-i-m: So for example, suppose I have the following:
|
||||
|
||||
macro_rules! foo { () => { println!(); } }
|
||||
|
||||
fn main() { foo!(); }
|
||||
|
||||
Then AST nodes that are finally generated would have parent(expn_id_println) ->
|
||||
parent(expn_id_foo), right?
|
||||
|
||||
Vadim Petrochenkov: Pretty common construction (at least it was, before
|
||||
refactorings) is SyntaxContext::empty().apply_mark(expn_id), which means...
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
Then AST nodes that are finally generated would have
|
||||
parent(expn_id_println) -> parent(expn_id_foo), right?
|
||||
|
||||
Yes.
|
||||
|
||||
mark-i-m:
|
||||
|
||||
and outer_expns are individual elements in the chain.
|
||||
|
||||
Sorry, what is outer_expns?
|
||||
|
||||
Vadim Petrochenkov: SyntaxContextData::outer_expn
|
||||
|
||||
mark-i-m: Thanks :) Please continue
|
||||
|
||||
Vadim Petrochenkov: ...which means a token produced by a built-in macro (which
|
||||
is defined in the root effectively).
|
||||
|
||||
mark-i-m: Where does the expn_id come from?
|
||||
|
||||
Vadim Petrochenkov: Or a stable proc macro, which are always considered to be
|
||||
defined in the root because they are always cross-crate, and we don't have the
|
||||
cross-crate hygiene implemented, ha-ha.
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
Where does the expn_id come from?
|
||||
|
||||
Vadim Petrochenkov: ID of the built-in macro call like line!().
|
||||
|
||||
Vadim Petrochenkov: Assigned continuously from 0 to N as soon as we discover
|
||||
new macro calls.
|
||||
|
||||
mark-i-m: Sorry, I didn't quite understand. Do you mean that only built-in
|
||||
macros receive continuous IDs?
|
||||
|
||||
Vadim Petrochenkov: So, the second hierarchy has a catch - the context
|
||||
transplantation hack -
|
||||
https://github.com/rust-lang/rust/pull/51762#issuecomment-401400732.
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
Do you mean that only built-in macros receive continuous IDs?
|
||||
|
||||
Vadim Petrochenkov: No, all macro calls receive ID.
|
||||
|
||||
Vadim Petrochenkov: Built-ins have the typical pattern
|
||||
SyntaxContext::empty().apply_mark(expn_id) for syntax contexts produced by
|
||||
them.
|
||||
|
||||
mark-i-m: I see, but this pattern is only used for built-ins, right?
|
||||
|
||||
Vadim Petrochenkov: And also all stable proc macros, see the comments above.
|
||||
|
||||
mark-i-m: Got it
|
||||
|
||||
Vadim Petrochenkov: The third hierarchy is call-site hierarchy.
|
||||
|
||||
Vadim Petrochenkov: If foo!(bar!(ident)) expands into ident
|
||||
|
||||
Vadim Petrochenkov: then hierarchy 1 is root -> foo -> bar -> ident
|
||||
|
||||
Vadim Petrochenkov: but hierarchy 3 is root -> ident
|
||||
|
||||
Vadim Petrochenkov: ExpnInfo::call_site is the child-parent link in this case.
|
||||
|
||||
mark-i-m: When we expand, do we expand foo first or bar? Why is there a
|
||||
hierarchy 1 here? Is that foo expands first and it expands to something that
|
||||
contains bar!(ident)?
|
||||
|
||||
Vadim Petrochenkov: Ah, yes, let's assume both foo and bar are identity macros.
|
||||
|
||||
Vadim Petrochenkov: Then foo!(bar!(ident)) -> expand -> bar!(ident) -> expand
|
||||
-> ident
|
||||
|
||||
Vadim Petrochenkov: If bar were expanded first, that would be eager expansion -
|
||||
https://github.com/rust-lang/rfcs/pull/2320.
|
||||
|
||||
mark-i-m: And after we expand only foo! presumably whatever intermediate state
|
||||
has heirarchy 1 of root->foo->(bar_ident), right?
|
||||
|
||||
Vadim Petrochenkov: (We have it hacked into some built-in macros, but not
|
||||
generally.)
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
And after we expand only foo! presumably whatever intermediate state has
|
||||
heirarchy 1 of root->foo->(bar_ident), right?
|
||||
|
||||
Vadim Petrochenkov: Yes.
|
||||
|
||||
mark-i-m: Got it :)
|
||||
|
||||
mark-i-m: It looks like we have ~5 minutes left. This has been very helpful
|
||||
already, but I also have more questions. Shall we try to schedule another
|
||||
meeting in the future?
|
||||
|
||||
Vadim Petrochenkov: Sure, why not.
|
||||
|
||||
Vadim Petrochenkov: A thread for offline questions-answers would be good too.
|
||||
|
||||
mark-i-m:
|
||||
|
||||
A thread for offline questions-answers would be good too.
|
||||
|
||||
I don't mind using this thread, since it already has a lot of info in it. We
|
||||
also plan to summarize the info from this thread into the rustc-guide.
|
||||
|
||||
Sure, why not.
|
||||
|
||||
Unfortunately, I'm unavailable for a few weeks. Would August 21-ish work for
|
||||
you (and @WG-learning )?
|
||||
|
||||
mark-i-m: @Vadim Petrochenkov Thanks very much for your time and knowledge!
|
||||
|
||||
mark-i-m: One last question: are there more hierarchies?
|
||||
|
||||
Vadim Petrochenkov: Not that I know of. Three + the context transplantation
|
||||
hack is already more complex than I'd like.
|
||||
|
||||
mark-i-m: Yes, one wonders what it would be like if one also had to think about
|
||||
eager expansion...
|
||||
|
||||
Santiago Pastorino: sorry but I couldn't follow that much today, will read it
|
||||
when I have some time later
|
||||
|
||||
Santiago Pastorino: btw https://github.com/rust-lang/rustc-guide/issues/398
|
||||
|
||||
mark-i-m: @Vadim Petrochenkov Would 7pm UTC on August 21 work for a followup?
|
||||
|
||||
Vadim Petrochenkov: Tentatively yes.
|
||||
|
||||
mark-i-m: @Vadim Petrochenkov @WG-learning Does this still work for everyone?
|
||||
|
||||
Vadim Petrochenkov: August 21 is still ok.
|
||||
|
||||
mark-i-m: @WG-learning @Vadim Petrochenkov We will start in ~30min
|
||||
|
||||
Vadim Petrochenkov: Oh. Thanks for the reminder, I forgot about this entirely.
|
||||
|
||||
mark-i-m: Hello!
|
||||
|
||||
Vadim Petrochenkov: (I'll be here in a couple of minutes.)
|
||||
|
||||
Vadim Petrochenkov: Ok, I'm here.
|
||||
|
||||
mark-i-m: Hi :)
|
||||
|
||||
Vadim Petrochenkov: Hi.
|
||||
|
||||
mark-i-m: so last time, we talked about the 3 context heirarchies
|
||||
|
||||
Vadim Petrochenkov: Right.
|
||||
|
||||
mark-i-m: Was there anything you wanted to add to that? If not, I think it
|
||||
would be good to get a big-picture... Given some piece of rust code, how do we
|
||||
get to the point where things are expanded and hygiene context is computed?
|
||||
|
||||
mark-i-m: (I'm assuming that hygiene info is computed as we expand stuff, since
|
||||
I don't think you can discover it beforehand)
|
||||
|
||||
Vadim Petrochenkov: Ok, let's move from hygiene to expansion.
|
||||
|
||||
Vadim Petrochenkov: Especially given that I don't remember the specific hygiene
|
||||
algorithms like adjust in detail.
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
Given some piece of rust code, how do we get to the point where things are
|
||||
expanded
|
||||
|
||||
So, first of all, the "some piece of rust code" is the whole crate.
|
||||
|
||||
mark-i-m: Just to confirm, the algorithms are well-encapsulated, right? Like a
|
||||
function or a struct as opposed to a bunch of conventions distributed across
|
||||
the codebase?
|
||||
|
||||
Vadim Petrochenkov: We run fully_expand_fragment in it.
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
Just to confirm, the algorithms are well-encapsulated, right?
|
||||
|
||||
Yes, the algorithmic parts are entirely inside hygiene.rs.
|
||||
|
||||
Vadim Petrochenkov: Ok, some are in fn resolve_crate_root, but those are hacks.
|
||||
|
||||
Vadim Petrochenkov: (Continuing about expansion.) If fully_expand_fragment is
|
||||
run not on a whole crate, it means that we are performing eager expansion.
|
||||
|
||||
Vadim Petrochenkov: Eager expansion is done for arguments of some built-in
|
||||
macros that expect literals.
|
||||
|
||||
Vadim Petrochenkov: It generally performs a subset of actions performed by the
|
||||
non-eager expansion.
|
||||
|
||||
Vadim Petrochenkov: So, I'll talk about non-eager expansion for now.
|
||||
|
||||
mark-i-m: Eager expansion is not exposed as a language feature, right? i.e. it
|
||||
is not possible for me to write an eager macro?
|
||||
|
||||
Vadim Petrochenkov:
|
||||
https://github.com/rust-lang/rust/pull/53778#issuecomment-419224049 (vvv The
|
||||
link is explained below vvv )
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
Eager expansion is not exposed as a language feature, right? i.e. it is not
|
||||
possible for me to write an eager macro?
|
||||
|
||||
Yes, it's entirely an ability of some built-in macros.
|
||||
|
||||
Vadim Petrochenkov: Not exposed for general use.
|
||||
|
||||
Vadim Petrochenkov: fully_expand_fragment works in iterations.
|
||||
|
||||
Vadim Petrochenkov: Iterations looks roughly like this:
|
||||
- Resolve imports in our partially built crate as much as possible.
|
||||
- Collect as many macro invocations as possible from our partially built crate
|
||||
(fn-like, attributes, derives) from the crate and add them to the queue.
|
||||
|
||||
Vadim Petrochenkov: Take a macro from the queue, and attempt to resolve it.
|
||||
|
||||
Vadim Petrochenkov: If it's resolved - run its expander function that
|
||||
consumes tokens or AST and produces tokens or AST (depending on the macro
|
||||
kind).
|
||||
|
||||
Vadim Petrochenkov: (If it's not resolved, then put it back into the
|
||||
queue.)
|
||||
|
||||
Vadim Petrochenkov: ^^^ That's where we fill in the hygiene data associated
|
||||
with ExpnIds.
|
||||
|
||||
mark-i-m: When we put it back in the queue?
|
||||
|
||||
mark-i-m: or do you mean the collect step in general?
|
||||
|
||||
Vadim Petrochenkov: Once we resolved the macro call to the macro definition we
|
||||
know everything about the macro and can call set_expn_data to fill in its
|
||||
properties in the global data.
|
||||
|
||||
Vadim Petrochenkov: I mean, immediately after successful resolution.
|
||||
|
||||
Vadim Petrochenkov: That's the first part of hygiene data, the second one is
|
||||
associated with SyntaxContext rather than with ExpnId, it's filled in later
|
||||
during expansion.
|
||||
|
||||
Vadim Petrochenkov: So, after we run the macro's expander function and got a
|
||||
piece of AST (or got tokens and parsed them into a piece of AST) we need to
|
||||
integrate that piece of AST into the big existing partially built AST.
|
||||
|
||||
Vadim Petrochenkov: This integration is a really important step where the next
|
||||
things happen:
|
||||
- NodeIds are assigned.
|
||||
|
||||
Vadim Petrochenkov: "def paths"s and their IDs (DefIds) are created
|
||||
|
||||
Vadim Petrochenkov: Names are put into modules from the resolver point of
|
||||
view.
|
||||
|
||||
Vadim Petrochenkov: So, we are basically turning some vague token-like mass
|
||||
into proper set in stone hierarhical AST and side tables.
|
||||
|
||||
Vadim Petrochenkov: Where exactly this happens - NodeIds are assigned by
|
||||
InvocationCollector (which also collects new macro calls from this new AST
|
||||
piece and adds them to the queue), DefIds are created by DefCollector, and
|
||||
modules are filled by BuildReducedGraphVisitor.
|
||||
|
||||
Vadim Petrochenkov: These three passes run one after another on every AST
|
||||
fragment freshly expanded from a macro.
|
||||
|
||||
Vadim Petrochenkov: After expanding a single macro and integrating its output
|
||||
we again try to resolve all imports in the crate, and then return to the big
|
||||
queue processing loop and pick up the next macro.
|
||||
|
||||
Vadim Petrochenkov: Repeat until there's no more macros. Vadim Petrochenkov:
|
||||
|
||||
mark-i-m: The integration step is where we would get parser errors too right?
|
||||
|
||||
mark-i-m: Also, when do we know definitively that resolution has failed for
|
||||
particular ident?
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
The integration step is where we would get parser errors too right?
|
||||
|
||||
Yes, if the macro produced tokens (rather than AST directly) and we had to
|
||||
parse them.
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
when do we know definitively that resolution has failed for particular
|
||||
ident?
|
||||
|
||||
So, ident is looked up in a number of scopes during resolution. From closest
|
||||
like the current block or module, to far away like preludes or built-in types.
|
||||
|
||||
Vadim Petrochenkov: If lookup is certainly failed in all of the scopes, then
|
||||
it's certainly failed.
|
||||
|
||||
mark-i-m: This is after all expansions and integrations are done, right?
|
||||
|
||||
Vadim Petrochenkov: "Certainly" is determined differently for different scopes,
|
||||
e.g. for a module scope it means no unexpanded macros and no unresolved glob
|
||||
imports in that module.
|
||||
|
||||
Vadim Petrochenkov:
|
||||
|
||||
This is after all expansions and integrations are done, right?
|
||||
|
||||
For macro and import names this happens during expansions and integrations.
|
||||
|
||||
mark-i-m: Makes sense
|
||||
|
||||
Vadim Petrochenkov: For all other names we certainly know whether a name is
|
||||
resolved successfully or not on the first attempt, because no new names can
|
||||
appear.
|
||||
|
||||
Vadim Petrochenkov: (They are resolved in a later pass, see
|
||||
librustc_resolve/late.rs.)
|
||||
|
||||
mark-i-m: And if at the end of the iteration, there are still things in the
|
||||
queue that can't be resolve, this represents an error, right?
|
||||
|
||||
mark-i-m: i.e. an undefined macro?
|
||||
|
||||
Vadim Petrochenkov: Yes, if we make no progress during an iteration, then we
|
||||
are stuck and that state represent an error.
|
||||
|
||||
Vadim Petrochenkov: We attempt to recover though, using dummies expanding into
|
||||
nothing or ExprKind::Err or something like that for unresolved macros.
|
||||
|
||||
mark-i-m: This is for the purposes of diagnostics, though, right?
|
||||
|
||||
Vadim Petrochenkov: But if we are going through recovery, then compilation must
|
||||
result in an error anyway.
|
||||
|
||||
Vadim Petrochenkov: Yes, that's for diagnostics, without recovery we would
|
||||
stuck at the first unresolved macro or import. Vadim Petrochenkov:
|
||||
|
||||
So, about the SyntaxContext hygiene...
|
||||
|
||||
Vadim Petrochenkov: New syntax contexts are created during macro expansion.
|
||||
|
||||
Vadim Petrochenkov: If the token had context X before being produced by a
|
||||
macro, e.g. here ident has context SyntaxContext::root(): Vadim Petrochenkov:
|
||||
|
||||
macro m() { ident }
|
||||
|
||||
Vadim Petrochenkov: , then after being produced by the macro it has context X
|
||||
-> macro_id.
|
||||
|
||||
Vadim Petrochenkov: I.e. our ident has context ROOT -> id(m) after it's
|
||||
produced by m.
|
||||
|
||||
Vadim Petrochenkov: The "chaining operator" -> is apply_mark in compiler code.
|
||||
Vadim Petrochenkov:
|
||||
|
||||
macro m() { macro n() { ident } }
|
||||
|
||||
Vadim Petrochenkov: In this example the ident has context ROOT originally, then
|
||||
ROOT -> id(m), then ROOT -> id(m) -> id(n).
|
||||
|
||||
Vadim Petrochenkov: Note that these chains are not entirely determined by their
|
||||
last element, in other words ExpnId is not isomorphic to SyntaxCtxt.
|
||||
|
||||
Vadim Petrochenkov: Couterexample: Vadim Petrochenkov:
|
||||
|
||||
macro m($i: ident) { macro n() { ($i, bar) } }
|
||||
|
||||
m!(foo);
|
||||
|
||||
Vadim Petrochenkov: foo has context ROOT -> id(n) and bar has context ROOT ->
|
||||
id(m) -> id(n) after all the expansions.
|
||||
|
||||
mark-i-m: Cool :)
|
||||
|
||||
mark-i-m: It looks like we are out of time
|
||||
|
||||
mark-i-m: Is there anything you wanted to add?
|
||||
|
||||
mark-i-m: We can schedule another meeting if you would like
|
||||
|
||||
Vadim Petrochenkov: Yep, 23.06 already. No, I think this is an ok point to
|
||||
stop.
|
||||
|
||||
mark-i-m: :+1:
|
||||
|
||||
mark-i-m: Thanks @Vadim Petrochenkov ! This was very helpful
|
||||
|
||||
Vadim Petrochenkov: Yeah, we can schedule another one. So far it's been like 1
|
||||
hour of meetings per month? Certainly not a big burden.
|
||||
```
|
||||
|
|
|
|||
Loading…
Reference in New Issue