From 4ba7b5da44df139d93a40ffc1df9ef2ca6b24263 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 11:35:37 -0400 Subject: [PATCH 1/9] document keep-stage1 --- src/how-to-build-and-run.md | 132 +++++++++++++++++++++++++++--------- 1 file changed, 99 insertions(+), 33 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 3ad9a52a..749340d7 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -53,24 +53,21 @@ One thing to keep in mind is that `rustc` is a _bootstrapping_ compiler. That is, since `rustc` is written in Rust, we need to use an older version of the compiler to compile the newer version. In particular, the newer version of the compiler, `libstd`, and other tooling may use some unstable features -internally. The result is the compiling `rustc` is done in stages. +internally. The result is the compiling `rustc` is done in stages: -- **Stage 0:** the stage0 compiler can be your existing - (perhaps older version of) - Rust compiler, the current _beta_ compiler or you may download the binary - from the internet. -- **Stage 1:** the code in your clone (for new version) - is then compiled with the stage0 - compiler to produce the stage1 compiler. - However, it was built with an older compiler (stage0), - so to optimize the stage1 compiler we go to next stage. -- **Stage 2:** we rebuild our stage1 compiler with itself - to produce the stage2 compiler (i.e. it builds - itself) to have all the _latest optimizations_. -- _(Optional)_ **Stage 3**: to sanity check of our new compiler, - we can build it again - with stage2 compiler which must be identical to itself, - unless something has broken. +- **Stage 0:** the stage0 compiler is usually the current _beta_ compiler + (`x.py` will download it for you); you can configure `x.py` to use something + else, though. +- **Stage 1:** the code in your clone (for new version) is then + compiled with the stage0 compiler to produce the stage1 compiler. + However, it was built with an older compiler (stage0), so to + optimize the stage1 compiler we go to next stage. +- **Stage 2:** we rebuild our stage1 compiler with itself to produce + the stage2 compiler (i.e. it builds itself) to have all the _latest + optimizations_. +- _(Optional)_ **Stage 3**: to sanity check of our new compiler, we + can build it again with stage2 compiler which must be identical to + itself, unless something has broken. For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. @@ -80,6 +77,8 @@ It is, in particular, very useful when you're doing some kind of "type-based refactoring", like renaming a method, or changing the signature of some function. + + Once you've created a config.toml, you are now ready to run `x.py`. There are a lot of options here, but let's start with what is probably the best "go to" command for building a local rust: @@ -88,27 +87,39 @@ probably the best "go to" command for building a local rust: > ./x.py build -i --stage 1 src/libstd ``` -What this command will do is the following: +This may *look* like it only builds libstd, but that is not the case. +What this command does is the following: -- Using the beta compiler (also called stage 0), it will build the - standard library and rustc from the `src` directory. The resulting - compiler is called the "stage 1" compiler. - - During this build, the `-i` (or `--incremental`) switch enables incremental - compilation, so that if you later rebuild after editing things in - `src`, you can save a bit of time. -- Using this stage 1 compiler, it will build the standard library. - (this is what the `src/libstd`) means. +- Build libstd using the stage0 compiler (using incremental) +- Build librustc using the stage0 compiler (using incremental) + - This produces the stage1 compiler +- Build libstd using the stage1 compiler (cannot use incremental) -This is just a subset of the full rustc build. The **full** rustc build -(what you get if you just say `./x.py build`) has quite a few more steps: +This final product (stage1 compiler + libs build using that compiler) +is what you need to build other rust programs. -- Build stage1 rustc with stage0 compiler. -- Build libstd with stage1 compiler (up to here is the same). -- Build rustc from `src` again, this time with the stage1 compiler - (this part is new). +Note that the command includes the `-i` switch. This enables incremental +compilation. This will be used to speed up the first two steps of the process: +in particular, if you make a small change, we ought to be able to use your old +results to make producing the stage1 **compiler** faster. + +Unfortunately, incremental cannot be used to speed up making the +stage1 libraries. This is because incremental only works when you run +the *same compiler* twice in a row. In this case, we are building a +*new stage1 compiler* every time. Therefore, the old incremental +results may not apply. **As a result, you will probably find that +building the stage1 libstd is a bottleneck for you** -- but fear not, +there is a (hacky) workaround. See [the section on "recommended +workflows"](#workflow) below. + +Note that this whole command just gives you a subset of the full rustc +build. The **full** rustc build (what you get if you just say `./x.py +build`) has quite a few more steps: + +- Build librustc rustc with the stage1 compiler. - The resulting compiler here is called the "stage2" compiler. - Build libstd with stage2 compiler. -- Build librustdoc and a bunch of other things. +- Build librustdoc and a bunch of other things with the stage2 compiler. @@ -148,6 +159,61 @@ release: 1.25.0-dev LLVM version: 4.0 ``` + + +### Suggested workflows for faster builds of the compiler + +There are two workflows that are useful for faster builders of the +compiler. + +**Check, check, and check again.** The first workflow, which is useful when doing +simple refactorings, is to run `./x.py check` continuously. Here you +are just checking that the compiler can **build**, but often that is +all you need (e.g., when renaming a method). You can then run `./x.py build` +when you acqtually need to run tests. + +In fact, it is eomtimes useful to put off tests even when you are not +100% sure the code will work. You can then keep building up +refactoring commits and only run the tests at some later time. You can +then use `git bisect` to track down **precisely** which commit caused +the problem. A nice side-effect of this style is that you are left +with a fairly fine-grained set of commits at the end, all of which +build and pass testes. This often helps reviewing. + +**Incremental builds with `--keep-stage`.** Sometimes just checking +whether the compiler builds is not enough. A common example is that +you need to add a `debug!` statement to inspect the value of some +state or better understand the problem. In that case, you really need +a full build. By leveraging incremental, though, you can often get +these builds to complete very fast (e.g., around 30 seconds): the only +catch is this requires a bit of fudging and may produce compilers that +don't work (but that is easily detected and fixed). + +The sequence of commands you want is as follows: + +- Initial build: `./x.py build -i --stage 1 src/libstd` + - As [documented above](#command), this will build a functional stage1 compiler +- Subsequent builds: `./x.py build -i --stage 1 src/libstd --keep-stage 1` + - Note that we added the `--keep-stage 1` flag here + +The effect of `--keep-stage1` is that we just *assume* that the old +standard library can be re-used. If you are editing the compiler, this +is almost always true: you haven't changed the standard library, after +all. But sometimes, it's not true: for example, if you are editing +the "metadata" part of the compiler, which controls how the compiler +encodes types and other states into the `rlib` files, or if you are editing +things that wind up in the metadata (such as the definition of the MIR). + +**The TL;DR is that you might get weird behavior from a compile when +using `--keep-stage 1`** -- for example, strange ICEs or other +panics. In that case, you should simply remove the `--keep-stage 1` +from the command and rebuild. That ought to fix the problem. + +Note: you can also use `--keep-stage 1` when running tests. Something like this: + +- Initial test run: `./x.py test -i --stage 1 src/test/ui` +- Subsequent test run: `./x.py test -i --stage 1 src/test/ui --keep-stage 1` + ### Other x.py commands Here are a few other useful x.py commands. We'll cover some of them in detail From 9af03dc02735a369f19a2915a15fafd8755fbe2c Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 12:46:10 -0400 Subject: [PATCH 2/9] wrap long lines --- src/how-to-build-and-run.md | 14 ++++++++------ 1 file changed, 8 insertions(+), 6 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 749340d7..59c31491 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -166,11 +166,12 @@ LLVM version: 4.0 There are two workflows that are useful for faster builders of the compiler. -**Check, check, and check again.** The first workflow, which is useful when doing -simple refactorings, is to run `./x.py check` continuously. Here you -are just checking that the compiler can **build**, but often that is -all you need (e.g., when renaming a method). You can then run `./x.py build` -when you acqtually need to run tests. +**Check, check, and check again.** The first workflow, which is useful +when doing simple refactorings, is to run `./x.py check` +continuously. Here you are just checking that the compiler can +**build**, but often that is all you need (e.g., when renaming a +method). You can then run `./x.py build` when you acqtually need to +run tests. In fact, it is eomtimes useful to put off tests even when you are not 100% sure the code will work. You can then keep building up @@ -192,7 +193,8 @@ don't work (but that is easily detected and fixed). The sequence of commands you want is as follows: - Initial build: `./x.py build -i --stage 1 src/libstd` - - As [documented above](#command), this will build a functional stage1 compiler + - As [documented above](#command), this will build a functional + stage1 compiler - Subsequent builds: `./x.py build -i --stage 1 src/libstd --keep-stage 1` - Note that we added the `--keep-stage 1` flag here From 93037428ec24fd8c3494cdfada2fc128cecca15b Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 12:48:54 -0400 Subject: [PATCH 3/9] note about why stage1/stage2 are different --- src/how-to-build-and-run.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 59c31491..f6c26ddf 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -62,6 +62,15 @@ internally. The result is the compiling `rustc` is done in stages: compiled with the stage0 compiler to produce the stage1 compiler. However, it was built with an older compiler (stage0), so to optimize the stage1 compiler we go to next stage. + - (In theory, the stage1 compiler is functionally identical to the + stage2 compiler, but in practice there are subtle differences. In + particular, the stage1 compiler itself was built by stage0 and + hence not by the source in your working directory: this means that + the symbol names used in the compiler source may not match the + symbol names that would have been made by the stage1 compiler. + This can be important when using dynamic linking (e.g., with + derives. Sometimes this means that some tests don't work when run + with stage1.) - **Stage 2:** we rebuild our stage1 compiler with itself to produce the stage2 compiler (i.e. it builds itself) to have all the _latest optimizations_. From 53eadd5f610c6fc27f1d5fffbb144bd46bcc5131 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 31 Aug 2018 12:49:16 -0400 Subject: [PATCH 4/9] fix typos --- src/how-to-build-and-run.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index f6c26ddf..aef5b66b 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -179,10 +179,10 @@ compiler. when doing simple refactorings, is to run `./x.py check` continuously. Here you are just checking that the compiler can **build**, but often that is all you need (e.g., when renaming a -method). You can then run `./x.py build` when you acqtually need to +method). You can then run `./x.py build` when you actually need to run tests. -In fact, it is eomtimes useful to put off tests even when you are not +In fact, it is sometimes useful to put off tests even when you are not 100% sure the code will work. You can then keep building up refactoring commits and only run the tests at some later time. You can then use `git bisect` to track down **precisely** which commit caused From 8927461dfc6f31d20670d6e67d502828801960ae Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:13:59 -0400 Subject: [PATCH 5/9] fix typo, add link --- src/how-to-build-and-run.md | 32 +++++++++++++++++--------------- 1 file changed, 17 insertions(+), 15 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index aef5b66b..2285b268 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -49,11 +49,12 @@ use-jemalloc = false ### Running x.py and building a stage1 compiler -One thing to keep in mind is that `rustc` is a _bootstrapping_ compiler. That -is, since `rustc` is written in Rust, we need to use an older version of the -compiler to compile the newer version. In particular, the newer version of the -compiler, `libstd`, and other tooling may use some unstable features -internally. The result is the compiling `rustc` is done in stages: +One thing to keep in mind is that `rustc` is a _bootstrapping_ +compiler. That is, since `rustc` is written in Rust, we need to use an +older version of the compiler to compile the newer version. In +particular, the newer version of the compiler, `libstd`, and other +tooling may use some unstable features internally. The result is that +compiling `rustc` is done in stages: - **Stage 0:** the stage0 compiler is usually the current _beta_ compiler (`x.py` will download it for you); you can configure `x.py` to use something @@ -73,10 +74,11 @@ internally. The result is the compiling `rustc` is done in stages: with stage1.) - **Stage 2:** we rebuild our stage1 compiler with itself to produce the stage2 compiler (i.e. it builds itself) to have all the _latest - optimizations_. + optimizations_. (By default, we copy the stage1 libraries for use by + the stage2 compiler, since they ought to be identical.) - _(Optional)_ **Stage 3**: to sanity check of our new compiler, we - can build it again with stage2 compiler which must be identical to - itself, unless something has broken. + can build the libraries with the stage2 compiler. The result ought + to be identical to before, unless something has broken. For hacking, often building the stage 1 compiler is enough, but for final testing and release, the stage 2 compiler is used. @@ -86,7 +88,7 @@ It is, in particular, very useful when you're doing some kind of "type-based refactoring", like renaming a method, or changing the signature of some function. - + Once you've created a config.toml, you are now ready to run `x.py`. There are a lot of options here, but let's start with what is @@ -104,8 +106,8 @@ What this command does is the following: - This produces the stage1 compiler - Build libstd using the stage1 compiler (cannot use incremental) -This final product (stage1 compiler + libs build using that compiler) -is what you need to build other rust programs. +This final product (stage1 compiler + libs built using that compiler) +is what you need to build other rust programs. Note that the command includes the `-i` switch. This enables incremental compilation. This will be used to speed up the first two steps of the process: @@ -125,7 +127,7 @@ Note that this whole command just gives you a subset of the full rustc build. The **full** rustc build (what you get if you just say `./x.py build`) has quite a few more steps: -- Build librustc rustc with the stage1 compiler. +- Build librustc and rustc with the stage1 compiler. - The resulting compiler here is called the "stage2" compiler. - Build libstd with stage2 compiler. - Build librustdoc and a bunch of other things with the stage2 compiler. @@ -168,7 +170,7 @@ release: 1.25.0-dev LLVM version: 4.0 ``` - + ### Suggested workflows for faster builds of the compiler @@ -188,7 +190,7 @@ refactoring commits and only run the tests at some later time. You can then use `git bisect` to track down **precisely** which commit caused the problem. A nice side-effect of this style is that you are left with a fairly fine-grained set of commits at the end, all of which -build and pass testes. This often helps reviewing. +build and pass tests. This often helps reviewing. **Incremental builds with `--keep-stage`.** Sometimes just checking whether the compiler builds is not enough. A common example is that @@ -216,7 +218,7 @@ encodes types and other states into the `rlib` files, or if you are editing things that wind up in the metadata (such as the definition of the MIR). **The TL;DR is that you might get weird behavior from a compile when -using `--keep-stage 1`** -- for example, strange ICEs or other +using `--keep-stage 1`** -- for example, strange [ICEs](appendix/glossary.html) or other panics. In that case, you should simply remove the `--keep-stage 1` from the command and rebuild. That ought to fix the problem. From b7976f5db2e1c90e3d3d0af0fe0e4eaa957de560 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:15:17 -0400 Subject: [PATCH 6/9] fix some more typos --- src/how-to-build-and-run.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 2285b268..17c883ff 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -209,13 +209,14 @@ The sequence of commands you want is as follows: - Subsequent builds: `./x.py build -i --stage 1 src/libstd --keep-stage 1` - Note that we added the `--keep-stage 1` flag here -The effect of `--keep-stage1` is that we just *assume* that the old +The effect of `--keep-stage 1` is that we just *assume* that the old standard library can be re-used. If you are editing the compiler, this is almost always true: you haven't changed the standard library, after all. But sometimes, it's not true: for example, if you are editing the "metadata" part of the compiler, which controls how the compiler -encodes types and other states into the `rlib` files, or if you are editing -things that wind up in the metadata (such as the definition of the MIR). +encodes types and other states into the `rlib` files, or if you are +editing things that wind up in the metadata (such as the definition of +the MIR). **The TL;DR is that you might get weird behavior from a compile when using `--keep-stage 1`** -- for example, strange [ICEs](appendix/glossary.html) or other From 016d366203c4a8811e89edfb461227372bc0015a Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Fri, 7 Sep 2018 11:16:20 -0400 Subject: [PATCH 7/9] remove silly `note:` --- src/how-to-build-and-run.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index 17c883ff..a4e6cc0f 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -223,7 +223,8 @@ using `--keep-stage 1`** -- for example, strange [ICEs](appendix/glossary.html) panics. In that case, you should simply remove the `--keep-stage 1` from the command and rebuild. That ought to fix the problem. -Note: you can also use `--keep-stage 1` when running tests. Something like this: +You can also use `--keep-stage 1` when running tests. Something like +this: - Initial test run: `./x.py test -i --stage 1 src/test/ui` - Subsequent test run: `./x.py test -i --stage 1 src/test/ui --keep-stage 1` From a9dab83da1dc26fd9c57d5cb8d4ff2d2b63dfaec Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 11 Sep 2018 12:02:46 -0400 Subject: [PATCH 8/9] wrap long lines --- src/how-to-build-and-run.md | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/src/how-to-build-and-run.md b/src/how-to-build-and-run.md index a4e6cc0f..66ba8efd 100644 --- a/src/how-to-build-and-run.md +++ b/src/how-to-build-and-run.md @@ -219,9 +219,10 @@ editing things that wind up in the metadata (such as the definition of the MIR). **The TL;DR is that you might get weird behavior from a compile when -using `--keep-stage 1`** -- for example, strange [ICEs](appendix/glossary.html) or other -panics. In that case, you should simply remove the `--keep-stage 1` -from the command and rebuild. That ought to fix the problem. +using `--keep-stage 1`** -- for example, strange +[ICEs](appendix/glossary.html) or other panics. In that case, you +should simply remove the `--keep-stage 1` from the command and +rebuild. That ought to fix the problem. You can also use `--keep-stage 1` when running tests. Something like this: From ca8c8774ef5ce823973c69b3db73e68a1533c471 Mon Sep 17 00:00:00 2001 From: Niko Matsakis Date: Tue, 11 Sep 2018 16:22:15 -0400 Subject: [PATCH 9/9] wrap lines in `running.md` --- src/tests/running.md | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/src/tests/running.md b/src/tests/running.md index fdc603a5..d410b79c 100644 --- a/src/tests/running.md +++ b/src/tests/running.md @@ -61,21 +61,24 @@ filtering for tests that include "issue-1234" in the name. ## Using incremental compilation -You can further enable the `--incremental` flag to save additional time in subsequent rebuilds: +You can further enable the `--incremental` flag to save additional +time in subsequent rebuilds: ```bash > ./x.py test --stage 1 src/test/ui --incremental --test-args issue-1234 ``` -If you don't want to include the flag with every command, you can enable it in the `config.toml`, too: +If you don't want to include the flag with every command, you can +enable it in the `config.toml`, too: ```toml # Whether to always use incremental compilation when building rustc incremental = true ``` -Note that incremental compilation will use more disk space than usual. If disk space is a -concern for you, you might want to check the size of the `build` directory from time to time. +Note that incremental compilation will use more disk space than +usual. If disk space is a concern for you, you might want to check the +size of the `build` directory from time to time. ## Running tests manually