[overview.md] add documentation of lexer support for Unicode encoding

This commit is contained in:
Chris Simpkins 2020-04-05 22:50:28 -04:00 committed by Who? Me?!
parent 18d695aa13
commit a12e9e31a3
1 changed files with 2 additions and 2 deletions

View File

@ -28,8 +28,8 @@ we'll talk about that later.
to the rest of the compilation process as a [`rustc_interface::Config`].
- The raw Rust source text is analyzed by a low-level lexer located in
[`librustc_lexer`]. At this stage, the source text is turned into a stream of
atomic source code units known as _tokens_. (**TODO**: chrissimpkins - Maybe
discuss Unicode handling during this stage?)
atomic source code units known as _tokens_. The lexer supports the Unicode
character encoding.
- The token stream passes through a higher-level lexer located in
[`librustc_parse`] to prepare for the next stage of the compile process. The
[`StringReader`] struct is used at this stage to perform a set of validations