document keep-stage1

src/how-to-build-and-run.md

@@ -53,24 +53,21 @@ One thing to keep in mind is that `rustc` is a _bootstrapping_ compiler. That
 is, since `rustc` is written in Rust, we need to use an older version of the
 compiler to compile the newer version. In particular, the newer version of the
 compiler, `libstd`, and other tooling may use some unstable features
-internally. The result is the compiling `rustc` is done in stages.
+internally. The result is the compiling `rustc` is done in stages:
 
-- **Stage 0:** the stage0 compiler can be your existing
-  (perhaps older version of)
-  Rust compiler, the current _beta_ compiler or you may download the binary
-  from the internet.
-- **Stage 1:** the code in your clone (for new version)
-  is then compiled with the stage0
-  compiler to produce the stage1 compiler.
-  However, it was built with an older compiler (stage0),
-  so to optimize the stage1 compiler we go to next stage.
-- **Stage 2:** we rebuild our stage1 compiler with itself
-  to produce the stage2 compiler (i.e. it builds
-  itself) to have all the _latest optimizations_.
-- _(Optional)_ **Stage 3**: to sanity check of our new compiler,
-  we can build it again
-  with stage2 compiler which must be identical to itself,
-  unless something has broken.
+- **Stage 0:** the stage0 compiler is usually the current _beta_ compiler
+  (`x.py` will download it for you); you can configure `x.py` to use something
+  else, though.
+- **Stage 1:** the code in your clone (for new version) is then
+  compiled with the stage0 compiler to produce the stage1 compiler.
+  However, it was built with an older compiler (stage0), so to
+  optimize the stage1 compiler we go to next stage.
+- **Stage 2:** we rebuild our stage1 compiler with itself to produce
+  the stage2 compiler (i.e. it builds itself) to have all the _latest
+  optimizations_.
+- _(Optional)_ **Stage 3**: to sanity check of our new compiler, we
+  can build it again with stage2 compiler which must be identical to
+  itself, unless something has broken.
 
 For hacking, often building the stage 1 compiler is enough, but for
 final testing and release, the stage 2 compiler is used.
@@ -80,6 +77,8 @@ It is, in particular, very useful when you're doing some kind of
 "type-based refactoring", like renaming a method, or changing the
 signature of some function.
 
+<a name=command>
+
 Once you've created a config.toml, you are now ready to run
 `x.py`. There are a lot of options here, but let's start with what is
 probably the best "go to" command for building a local rust:
@@ -88,27 +87,39 @@ probably the best "go to" command for building a local rust:
 > ./x.py build -i --stage 1 src/libstd
 ```
 
-What this command will do is the following:
+This may *look* like it only builds libstd, but that is not the case.
+What this command does is the following:
 
-- Using the beta compiler (also called stage 0), it will build the
-  standard library and rustc from the `src` directory. The resulting
-  compiler is called the "stage 1" compiler.
-  - During this build, the `-i` (or `--incremental`) switch enables incremental
-    compilation, so that if you later rebuild after editing things in
-    `src`, you can save a bit of time.
-- Using this stage 1 compiler, it will build the standard library.
-  (this is what the `src/libstd`) means.
+- Build libstd using the stage0 compiler (using incremental)
+- Build librustc using the stage0 compiler (using incremental)
+  - This produces the stage1 compiler
+- Build libstd using the stage1 compiler (cannot use incremental)
 
-This is just a subset of the full rustc build. The **full** rustc build
-(what you get if you just say `./x.py build`) has quite a few more steps:
+This final product (stage1 compiler + libs build using that compiler)
+is what you need to build other rust programs. 
 
-- Build stage1 rustc with stage0 compiler.
-- Build libstd with stage1 compiler (up to here is the same).
-- Build rustc from `src` again, this time with the stage1 compiler
-  (this part is new).
+Note that the command includes the `-i` switch. This enables incremental
+compilation. This will be used to speed up the first two steps of the process:
+in particular, if you make a small change, we ought to be able to use your old
+results to make producing the stage1 **compiler** faster.
+
+Unfortunately, incremental cannot be used to speed up making the
+stage1 libraries.  This is because incremental only works when you run
+the *same compiler* twice in a row.  In this case, we are building a
+*new stage1 compiler* every time. Therefore, the old incremental
+results may not apply. **As a result, you will probably find that
+building the stage1 libstd is a bottleneck for you** -- but fear not,
+there is a (hacky) workaround.  See [the section on "recommended
+workflows"](#workflow) below.
+
+Note that this whole command just gives you a subset of the full rustc
+build. The **full** rustc build (what you get if you just say `./x.py
+build`) has quite a few more steps:
+
+- Build librustc rustc with the stage1 compiler.
   - The resulting compiler here is called the "stage2" compiler.
 - Build libstd with stage2 compiler.
-- Build librustdoc and a bunch of other things.
+- Build librustdoc and a bunch of other things with the stage2 compiler.
 
 <a name=toolchain></a>
 
@@ -148,6 +159,61 @@ release: 1.25.0-dev
 LLVM version: 4.0
 ```
 
+<a name=workflow>
+
+### Suggested workflows for faster builds of the compiler
+
+There are two workflows that are useful for faster builders of the
+compiler.
+
+**Check, check, and check again.** The first workflow, which is useful when doing
+simple refactorings, is to run `./x.py check` continuously. Here you
+are just checking that the compiler can **build**, but often that is
+all you need (e.g., when renaming a method). You can then run `./x.py build`
+when you acqtually need to run tests.
+
+In fact, it is eomtimes useful to put off tests even when you are not
+100% sure the code will work. You can then keep building up
+refactoring commits and only run the tests at some later time. You can
+then use `git bisect` to track down **precisely** which commit caused
+the problem. A nice side-effect of this style is that you are left
+with a fairly fine-grained set of commits at the end, all of which
+build and pass testes. This often helps reviewing.
+
+**Incremental builds with `--keep-stage`.** Sometimes just checking
+whether the compiler builds is not enough. A common example is that
+you need to add a `debug!` statement to inspect the value of some
+state or better understand the problem. In that case, you really need
+a full build.  By leveraging incremental, though, you can often get
+these builds to complete very fast (e.g., around 30 seconds): the only
+catch is this requires a bit of fudging and may produce compilers that
+don't work (but that is easily detected and fixed).
+
+The sequence of commands you want is as follows:
+
+- Initial build: `./x.py build -i --stage 1 src/libstd`
+  - As [documented above](#command), this will build a functional stage1 compiler
+- Subsequent builds: `./x.py build -i --stage 1 src/libstd --keep-stage 1`
+  - Note that we added the `--keep-stage 1` flag here
+  
+The effect of `--keep-stage1` is that we just *assume* that the old
+standard library can be re-used. If you are editing the compiler, this
+is almost always true: you haven't changed the standard library, after
+all.  But sometimes, it's not true: for example, if you are editing
+the "metadata" part of the compiler, which controls how the compiler
+encodes types and other states into the `rlib` files, or if you are editing
+things that wind up in the metadata (such as the definition of the MIR).
+
+**The TL;DR is that you might get weird behavior from a compile when
+using `--keep-stage 1`** -- for example, strange ICEs or other
+panics. In that case, you should simply remove the `--keep-stage 1`
+from the command and rebuild.  That ought to fix the problem.
+
+Note: you can also use `--keep-stage 1` when running tests. Something like this:
+
+- Initial test run: `./x.py test -i --stage 1 src/test/ui`
+- Subsequent test run: `./x.py test -i --stage 1 src/test/ui --keep-stage 1`
+
 ### Other x.py commands
 
 Here are a few other useful x.py commands. We'll cover some of them in detail