From 1424834cd19171237ca9675bece2a22b0929633d Mon Sep 17 00:00:00 2001 From: mark Date: Sat, 4 Apr 2020 12:52:55 -0500 Subject: [PATCH] mention token stream as an IR --- src/overview.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/src/overview.md b/src/overview.md index 0648ac7e..cebff545 100644 --- a/src/overview.md +++ b/src/overview.md @@ -149,8 +149,11 @@ Instead most compilers, including `rustc`, build some sort of IR out of the source code which is easier to analyze. `rustc` has a few IRs, each optimized for different purposes: +- Token stream: the lexer produces a stream of tokens directly from the source + code. This stream of tokens is easier for the parser to deal with than raw + text. - Abstract Syntax Tree (AST): the abstract syntax tree is built from the stream - of tokens produced by the lexer directly from the source code. It represents + of tokens produced by the lexer. It represents pretty much exactly what the user wrote. It helps to do some syntactic sanity checking (e.g. checking that a type is expected where the user wrote one). - High-level IR (HIR): This is a sort of desugared AST. It's still close