go/src/cmd/dist
Russ Cox 8a27154bcd cmd/dist: make toolchain build reproducible
- Build cmd with CGO_ENABLED=0. Doing so removes the C compiler
  toolchain from the reproducibility perimeter and also results in
  cmd/go and cmd/pprof binaries that are statically linked,
  so that they will run on a wider variety of systems.
  In particular the Linux versions will run on Alpine and NixOS
  without needing a simulation of libc.so.6.

  The potential downside of disabling cgo is that cmd/go and cmd/pprof
  use the pure Go network resolver instead of the host resolver on
  Unix systems. This means they will not be able to use non-DNS
  resolver mechanisms that may be specified in /etc/resolv.conf,
  such as mDNS. Neither program seems likely to need non-DNS names
  like those, however.

  macOS and Windows systems still use the host resolver, which they
  access without cgo.

- Build cmd with -trimpath when building a release.
  Doing so removes $GOPATH from the file name prefixes stored in the
  binary, so that the build directory does not leak into the final artifacts.

- When CC and CXX are empty, do not pick values to hard-code into
  the source tree and binaries. Instead, emit code that makes the
  right decision at runtime. In addition to reproducibility, this
  makes cross-compiled toolchains work better. A macOS toolchain
  cross-compiled on Linux will now correctly look for clang,
  instead of looking for gcc because it was built on Linux.

- Convert \ to / in file names stored in .a files.
  These are converted to / in the final binaries, but the hashes of
  the .a files affect the final build ID of the binaries. Without this
  change, builds of a Windows toolchain on Windows and non-Windows
  machines produce identical binaries except for the input hash part
  of the build ID.

- Due to the conversion of \ to / in .a files, convert back when
  reading inline bodies on Windows to preserve output file names
  in error messages.

Combined, these four changes (along with Go 1.20's removal of
installed pkg/**.a files and conversion of macOS net away from cgo)
make the output of make.bash fully reproducible, even when
cross-compiling: a released macOS toolchain built on Linux or Windows
will contain exactly the same bits as a released macOS toolchain
built on macOS.

The word "released" in the previous sentence is important.
For the build IDs in the binaries to work out the same on
both systems, a VERSION file must exist to provide a consistent
compiler build ID (instead of using a content hash of the binary).

For #24904.
Fixes #57007.

Change-Id: I665e1ef4ff207d6ff469452347dca5bfc81050e6
Reviewed-on: https://go-review.googlesource.com/c/go/+/454836
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
Auto-Submit: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2023-01-17 23:10:31 +00:00
..
README cmd/dist: update to assume Go 1.17.13 as minimal Go bootstrap version 2022-11-21 17:16:41 +00:00
build.go cmd/dist: make toolchain build reproducible 2023-01-17 23:10:31 +00:00
buildgo.go cmd/dist: make toolchain build reproducible 2023-01-17 23:10:31 +00:00
buildruntime.go cmd/dist: use strings.Builder 2022-09-08 14:33:38 +00:00
buildtag.go cmd/dist: implement //go:build parsing 2021-10-28 03:35:04 +00:00
buildtag_test.go cmd/dist: add buildtag parsing test 2021-11-25 00:02:52 +00:00
buildtool.go cmd/dist: update to assume Go 1.17.13 as minimal Go bootstrap version 2022-11-21 17:16:41 +00:00
doc.go all: gofmt main repo 2022-04-11 16:34:30 +00:00
exec_118.go cmd/dist: add old style build tags 2022-09-17 02:56:44 +00:00
exec_119.go cmd/dist: add old style build tags 2022-09-17 02:56:44 +00:00
imports.go cmd/dist: support GOROOT vendoring 2019-10-28 21:45:30 +00:00
main.go cmd/dist: make toolchain build reproducible 2023-01-17 23:10:31 +00:00
notgo117.go all: use Go 1.17.13 for bootstrap 2022-10-17 19:46:59 +00:00
quoted.go all: add missing copyright header 2022-11-09 14:05:53 +00:00
sys_default.go all: go fmt std cmd (but revert vendor) 2021-02-20 03:54:50 +00:00
sys_windows.go Revert "cmd/dist: log CPU model when testing" 2022-01-14 20:50:36 +00:00
test.go cmd/dist: make toolchain build reproducible 2023-01-17 23:10:31 +00:00
util.go cmd/dist: make toolchain build reproducible 2023-01-17 23:10:31 +00:00
util_gc.go all: use more precise build tags 2021-02-23 19:55:03 +00:00
util_gccgo.go all: go fmt std cmd (but revert vendor) 2021-02-20 03:54:50 +00:00
vfp_arm.s all: add //go:build lines to assembly files 2021-05-13 09:12:17 +00:00
vfp_default.s all: add //go:build lines to assembly files 2021-05-13 09:12:17 +00:00

README

This program, dist, is the bootstrapping tool for the Go distribution.

As of Go 1.5, dist and other parts of the compiler toolchain are written
in Go, making bootstrapping a little more involved than in the past.
The approach is to build the current release of Go with an earlier one.

The process to install Go 1.x, for x ≥ 20, is:

1. Build cmd/dist with Go 1.17.13.
2. Using dist, build Go 1.x compiler toolchain with Go 1.17.13.
3. Using dist, rebuild Go 1.x compiler toolchain with itself.
4. Using dist, build Go 1.x cmd/go (as go_bootstrap) with Go 1.x compiler toolchain.
5. Using go_bootstrap, build the remaining Go 1.x standard library and commands.

NOTE: During the transition from the old C-based toolchain to the Go-based one,
step 2 also builds the parts of the toolchain written in C, and step 3 does not
recompile those.

Because of backward compatibility, although the steps above say Go 1.17.13,
in practice any release ≥ Go 1.17.13 but < Go 1.x will work as the bootstrap base.

See golang.org/s/go15bootstrap for more details.

Compared to Go 1.4 and earlier, dist will also take over much of what used to
be done by make.bash/make.bat/make.rc and all of what used to be done by
run.bash/run.bat/run.rc, because it is nicer to implement that logic in Go
than in three different scripting languages simultaneously.