mirror
/
rustc-dev-guide
mirror of https://github.com/rust-lang/rustc-dev-guide.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

write a short paragraph about the new lexer

This commit is contained in:
Aleksey Kladov 2019-07-21 14:33:57 +03:00 committed by Who? Me?!
parent 831a469050
commit d8af1f7fff
1 changed files with 13 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

13
src/the-parser.md
View File

@ -32,6 +32,19 @@ To minimise the amount of copying that is done, both the `StringReader` and
`Parser` have lifetimes which bind them to the parent `ParseSess`. This contains
all the information needed while parsing, as well as the `SourceMap` itself.
## More on Lexical Analysis
Code for lexical analysis is split between two crates:
- `rustc_lexer` crate is responsible for breaking a `&str` into chunks
constituting tokens. Although it is popular to implement lexers as generated
finite state machines, the lexer in `rustc_lexer` is hand-written.
- [`StringReader`] from [libsyntax] integrates `rustc_lexer` with `rustc`
specific data structures. Specifically, it adds `Span` information to tokens
returned by `rustc_lexer` and interns identifiers.
[libsyntax]: https://doc.rust-lang.org/nightly/nightly-rustc/syntax/index.html
[rustc_errors]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html
[ast]: https://en.wikipedia.org/wiki/Abstract_syntax_tree