From 10c6b7a7490d2b4397e1a9710d20456f1ed311bf Mon Sep 17 00:00:00 2001 From: mark Date: Wed, 6 May 2020 13:59:28 -0500 Subject: [PATCH] add a bit more on parallel compilation --- src/SUMMARY.md | 1 + src/parallel-rustc.md | 45 +++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 46 insertions(+) create mode 100644 src/parallel-rustc.md diff --git a/src/SUMMARY.md b/src/SUMMARY.md index fe040d9e..f1efcf59 100644 --- a/src/SUMMARY.md +++ b/src/SUMMARY.md @@ -44,6 +44,7 @@ - [Profiling Queries](./queries/profiling.md) - [Salsa](./salsa.md) - [Memory Management in Rustc](./memory.md) + - [Parallel Compilation](./parallel-rustc.md) - [Part 3: Source Code Representations](./part-3-intro.md) - [The Rustc Driver and Interface](./rustc-driver.md) diff --git a/src/parallel-rustc.md b/src/parallel-rustc.md new file mode 100644 index 00000000..c8214af3 --- /dev/null +++ b/src/parallel-rustc.md @@ -0,0 +1,45 @@ +# Parallel Compilation + +Most of the compiler is not parallel. This represents an opportunity for +improving compiler performance. Much effort has been put into parallelizing +`rustc`, but it is still pretty early days for this work. There is a lot of +design and correctness work that needs to be done. + +One can try out the current parallel compiler work by enabling it in the +`config.toml`. + +There are a few basic ideas in this effort: + +- There are a lot of loops in the compiler that just iterate over all items in + a crate. These can possibly be parallelized. +- We can use (a custom fork of) [`rayon`] to run tasks in parallel. The custom + fork allows the execution of DAGs of tasks, not just trees. +- There are currently a lot of global data structures that need to be made + thread-safe. A key strategy here has been converting interior-mutable + data-structures (e.g. `Cell`) into their thread-safe siblings (e.g. `Mutex`). + +[`rayon`]: https://crates.io/rayon + +As of this writing, much of this effort is on hold due to lack of manpower. We +have a working prototype with promising performance gains in many cases. +However, there are two blockers: + +- It's not clear what invariants need to be upheld that might not hold in the + face of concurrency. An auditing effort was underway, but seems to have + stalled at some point. + +- There is a lot of lock contention, which actually degrades performance as the + number of threads increases beyond 4. + +Here are some resources that can used to learn more (note that some of them are +a bit out of date): + +- [This IRLO thread by Zoxc, when of the pioneers of the effort][irlo0] +- [This list of interior mutability in the compiler by nikomatsakis][imlist] +- [This IRLO thread by alexchricton about performance][irlo1] +- [This tracking issue][tracking] + +[irlo0]: https://internals.rust-lang.org/t/parallelizing-rustc-using-rayon/6606 +[imlist]: https://github.com/nikomatsakis/rustc-parallelization/blob/master/interior-mutability-list.md +[irlo1]: https://internals.rust-lang.org/t/help-test-parallel-rustc/11503 +[tracking]: https://github.com/rust-lang/rust/issues/48685