# High-level overview of the compiler source

> **NOTE**: The structure of the repository is going through a lot of
> transitions. In particular, we want to get to a point eventually where the
> top-level directory has separate directories for the compiler, build-system,
> std libs, etc, rather than one huge `src/` directory.

## Workspace structure

The `rust-lang/rust` repository consists of a single large cargo workspace
containing the compiler, the standard library (core, alloc, std, etc), and
`rustdoc`, along with the build system and bunch of tools and submodules for
building a full Rust distribution.

As of this writing, this structure is gradually undergoing some transformation
to make it a bit less monolithic and more approachable, especially to
newcommers.

> Eventually, the hope is for the standard library to live in a `stdlib/`
> directory, while the compiler lives in `compiler/`. However, as of this
> writing, both live in `src/`.

The repository consists of a `src` directory, under which there live many
crates, which are the source for the compiler, standard library, etc, as
mentioned above.

## Standard library

The standard library crates are obviously named `libstd`, `libcore`,
`liballoc`, etc. There is also `libproc_macro`, `libtest`, and other runtime
libraries.

This code is fairly similar to most other Rust crates except that it must be
built in a special way because it can use unstable features.

## Compiler

The compiler crates all have names starting with `librustc_*`. These are a large
collection of interdependent crates. There is also the `rustc` crate which is
the actual binary. It doesn't actually do anything besides calling the compiler
main function elsewhere.

The dependency structure of these crates is complex, but roughly it is
something like this:

- `rustc` (the binary) calls [`rustc_driver::main`][main].
    - [`rustc_driver`] depends on a lot of other crates, but the main one is
      [`rustc_interface`].
        - [`rustc_interface`] depends on most of the other compiler crates. It
          is a fairly generic interface for driving the whole compilation.
            - The most of the other `rustc_*` crates depend on [`rustc_middle`],
              which defines a lot of central data structures in the compiler.
                - [`rustc_middle`] and most of the other crates depend on a
                  handful of crates representing the early parts of the
                  compiler (e.g. the parser), fundamental data structures (e.g.
                  [`Span`]), or error reporting: [`rustc_data_structures`],
                  [`rustc_span`], [`rustc_errors`], etc.

[main]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/fn.main.html
[`rustc_driver`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_driver/index.html
[`rustc_interface`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_interface/index.html
[`rustc_middle`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_middle/index.html
[`rustc_data_structures`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_data_structures/index.html
[`rustc_span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/index.html
[`Span`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_span/struct.Span.html
[`rustc_errors`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustc_errors/index.html

You can see the exact dependencies by reading the `Cargo.toml` for the various
crates, just like a normal Rust crate.

One final thing: [`src/llvm-project`] is a submodule for our fork of LLVM.

Most of this book is about the compiler, so we won't have any further
explanation of these crates here.

[`src/llvm-project`]: https://github.com/rust-lang/rust/tree/master/src

### Big picture

The dependency structure is influenced strongly by two main factors:

1. Organization. The compiler is a _huge_ codebase; it would be an impossibly
   large crate. In part, the dependency structure reflects the code structure
   of the compiler.
2. Compile time. By breaking the compiler into multiple crates, we can take
   better advantage of incremental/parallel compilation using cargo. In
   particular, we try to have as few dependencies between crates as possible so
   that we dont' have to rebuild as many crates if you change one.

At the very bottom of the dependency tree are a handful of crates that are used
by the whole compiler (e.g. [`rustc_span`]). The very early parts of the
compilation process (e.g. parsing and the AST) depend on only these.

Pretty soon after the AST is constructed, the compiler's [query system][query]
gets set up.  The query system is set up in a clever way using function
pointers. This allows us to break dependencies between crates, allowing more
parallel compilation.

However, since the query system is defined in [`rustc_middle`], nearly all
subsequent parts of the compiler depend on this crate. It is a really large
crate, leading to long compile times. Some efforts have been made to move stuff
out of it with limited success. Another unfortunate sideffect is that sometimes
related functionality gets scattered across different crates. For example,
linting functionality is scattered across earlier parts of the crate,
[`rustc_lint`], [`rustc_middle`], and other places.

More generally, in an ideal world, it seems like there would be fewer, more
cohesive crates, with incremental and parallel compilation making sure compile
times stay reasonable. However, our incremental and parallel compilation haven't
gotten good enough for that yet, so breaking things into separate crates has
been our solution so far.

At the top of the dependency tree are the [`rustc_interface`] and
[`rustc_driver`] crates. [`rustc_interface`] is an unstable wrapper around the
query system that helps to drive the various stages of compilation. Other
consumers of the compiler may use this interface in different ways (e.g.
rustdoc or maybe eventually rust-analyzer). The [`rustc_driver`] crate first
parses command line arguments and then uses [`rustc_interface`] to drive the
compilation to completion.

[query]: ./query.md

[orgch]: ./overview.md

## rustdoc

The bulk of `rustdoc` is in [`librustdoc`]. However, the `rustdoc` binary
itself is [`src/tools/rustdoc`], which does nothing except call [`rustdoc::main`].

There is also javascript and CSS for the rustdocs in [`src/tools/rustdoc-js`]
and [`src/tools/rustdoc-themes`].

You can read more about rustdoc in [this chapter][rustdocch].

[`librustdoc`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/index.html
[`rustdoc::main`]: https://doc.rust-lang.org/nightly/nightly-rustc/rustdoc/fn.main.html
[`src/tools/rustdoc`]:  https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc
[`src/tools/rustdoc-js`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-js
[`src/tools/rustdoc-themes`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustdoc-themes

[rustdocch]: ./rustdoc-internals.md

## Tests

The test suite for all of the above is in [`src/test/`]. You can read more
about the test suite [in this chapter][testsch].

The test harness itself is in [`src/tools/compiletest`].

[testsch]: ./tests/intro.md

[`src/test/`]: https://github.com/rust-lang/rust/tree/master/src/test
[`src/tools/compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest

## Build System

There are a number of tools in the repository just for building the compiler,
standard library, rustdoc, etc, along with testing, building a full Rust
distribution, etc.

One of the primary tools is [`src/bootstrap`]. You can read more about
bootstrapping [in this chapter][bootstch]. The process may also use other tools
from `src/tools/`, such as [`tidy`] or [`compiletest`].

[`src/bootstrap`]: https://github.com/rust-lang/rust/tree/master/src/bootstrap
[`tidy`]: https://github.com/rust-lang/rust/tree/master/src/tools/tidy
[`compiletest`]: https://github.com/rust-lang/rust/tree/master/src/tools/compiletest

[bootstch]: ./building/bootstrapping.md

## Other

There are a lot of other things in the `rust-lang/rust` repo that are related
to building a full rust distribution. Most of the time you don't need to worry
about them.

These include:
- [`src/ci`]: The CI configuration. This actually quite extensive because we
  run a lot of tests on a lot of platforms.
- [`src/doc`]: Various documentation, including submodules for a few books.
- [`src/etc`]: Miscellaneous utilities.
- [`src/tools/rustc-workspace-hack`], and others: Various workarounds to make
  cargo work with bootstrapping.
- And more...

[`src/ci`]: https://github.com/rust-lang/rust/tree/master/src/ci
[`src/doc`]: https://github.com/rust-lang/rust/tree/master/src/doc
[`src/etc`]: https://github.com/rust-lang/rust/tree/master/src/etc
[`src/tools/rustc-workspace-hack`]: https://github.com/rust-lang/rust/tree/master/src/tools/rustc-workspace-hack