mirror
/
rustc-dev-guide
mirror of https://github.com/rust-lang/rustc-dev-guide.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request #2247 from marcoieni/ci-cache 

improve CI cache docs
This commit is contained in:
Marco Ieni 2025-02-06 15:01:30 +01:00 committed by GitHub
parent 03a1548373 8a6fd47a85
commit 4e55c24537
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
1 changed files with 20 additions and 4 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

24
src/tests/ci.md
View File

@ -322,7 +322,7 @@ Our CI workflow uses various caching mechanisms, mainly for two things:
### Docker images caching ### Docker images caching
The Docker images we use to run most of the Linux-based builders take a *long* The Docker images we use to run most of the Linux-based builders take a *long*
time to fully build. To speed up the build, we cache it using [Docker registry time to fully build. To speed up the build, we cache them using [Docker registry
caching], with the intermediate artifacts being stored on [ghcr.io]. We also caching], with the intermediate artifacts being stored on [ghcr.io]. We also
push the built Docker images to ghcr, so that they can be reused by other tools push the built Docker images to ghcr, so that they can be reused by other tools
(rustup) or by developers running the Docker build locally (to speed up their (rustup) or by developers running the Docker build locally (to speed up their
@ -334,6 +334,13 @@ override the cache for the others. Instead, we store the images under different
tags, identifying them with a custom hash made from the contents of all the tags, identifying them with a custom hash made from the contents of all the
Dockerfiles and related scripts. Dockerfiles and related scripts.
The CI calculates a hash key, so that the cache of a Docker image is
invalidated if one of the following changes:
- Dockerfile
- Files copied into the Docker image in the Dockerfile
- The architecture of the GitHub runner (x86 or ARM)
[ghcr.io]: https://github.com/rust-lang-ci/rust/pkgs/container/rust-ci [ghcr.io]: https://github.com/rust-lang-ci/rust/pkgs/container/rust-ci
[Docker registry caching]: https://docs.docker.com/build/cache/backends/registry/ [Docker registry caching]: https://docs.docker.com/build/cache/backends/registry/
@ -341,9 +348,18 @@ Dockerfiles and related scripts.
We build some C/C++ stuff in various CI jobs, and we rely on [sccache] to cache We build some C/C++ stuff in various CI jobs, and we rely on [sccache] to cache
the intermediate LLVM artifacts. Sccache is a distributed ccache developed by the intermediate LLVM artifacts. Sccache is a distributed ccache developed by
Mozilla, which can use an object storage bucket as the storage backend. In our Mozilla, which can use an object storage bucket as the storage backend.
case, the artefacts are uploaded to an S3 bucket that we control
(`rust-lang-ci-sccache2`). With sccache there's no need to calculate the hash key ourselves. Sccache
invalidates the cache automatically when it detects changes to relevant inputs,
such as the source code, the version of the compiler, and important environment
variables.
So we just pass the sccache wrapper on top of cargo and sccache does the rest.
We store the persistent artifacts on the S3 bucket `rust-lang-ci-sccache2`. So
when the CI runs, if sccache sees that LLVM is being compiled with the same C/C++
compiler and the LLVM source code is the same, sccache retrieves the individual
compiled translation units from S3.
[sccache]: https://github.com/mozilla/sccache [sccache]: https://github.com/mozilla/sccache