go/src
Russ Cox 8a27154bcd cmd/dist: make toolchain build reproducible
- Build cmd with CGO_ENABLED=0. Doing so removes the C compiler
  toolchain from the reproducibility perimeter and also results in
  cmd/go and cmd/pprof binaries that are statically linked,
  so that they will run on a wider variety of systems.
  In particular the Linux versions will run on Alpine and NixOS
  without needing a simulation of libc.so.6.

  The potential downside of disabling cgo is that cmd/go and cmd/pprof
  use the pure Go network resolver instead of the host resolver on
  Unix systems. This means they will not be able to use non-DNS
  resolver mechanisms that may be specified in /etc/resolv.conf,
  such as mDNS. Neither program seems likely to need non-DNS names
  like those, however.

  macOS and Windows systems still use the host resolver, which they
  access without cgo.

- Build cmd with -trimpath when building a release.
  Doing so removes $GOPATH from the file name prefixes stored in the
  binary, so that the build directory does not leak into the final artifacts.

- When CC and CXX are empty, do not pick values to hard-code into
  the source tree and binaries. Instead, emit code that makes the
  right decision at runtime. In addition to reproducibility, this
  makes cross-compiled toolchains work better. A macOS toolchain
  cross-compiled on Linux will now correctly look for clang,
  instead of looking for gcc because it was built on Linux.

- Convert \ to / in file names stored in .a files.
  These are converted to / in the final binaries, but the hashes of
  the .a files affect the final build ID of the binaries. Without this
  change, builds of a Windows toolchain on Windows and non-Windows
  machines produce identical binaries except for the input hash part
  of the build ID.

- Due to the conversion of \ to / in .a files, convert back when
  reading inline bodies on Windows to preserve output file names
  in error messages.

Combined, these four changes (along with Go 1.20's removal of
installed pkg/**.a files and conversion of macOS net away from cgo)
make the output of make.bash fully reproducible, even when
cross-compiling: a released macOS toolchain built on Linux or Windows
will contain exactly the same bits as a released macOS toolchain
built on macOS.

The word "released" in the previous sentence is important.
For the build IDs in the binaries to work out the same on
both systems, a VERSION file must exist to provide a consistent
compiler build ID (instead of using a content hash of the binary).

For #24904.
Fixes #57007.

Change-Id: I665e1ef4ff207d6ff469452347dca5bfc81050e6
Reviewed-on: https://go-review.googlesource.com/c/go/+/454836
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Russ Cox <rsc@golang.org>
Auto-Submit: Russ Cox <rsc@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2023-01-17 23:10:31 +00:00
..
archive archive/tar, archive/zip: document ErrInsecurePath and GODEBUG setting 2023-01-17 14:18:16 +00:00
arena
bufio
builtin
bytes
cmd cmd/dist: make toolchain build reproducible 2023-01-17 23:10:31 +00:00
compress
container
context
crypto crypto/ed25519: improve Ed25519ctx docs and add example 2023-01-17 16:25:39 +00:00
database/sql
debug debug/buildinfo: check pointer size on buildinfo.Read 2022-12-12 14:16:49 +00:00
embed
encoding encoding/csv: use proper doc comment for Deprecated notes 2022-12-02 16:30:23 +00:00
errors
expvar
flag
fmt
go go/types, types2: more uses of factored functions; generate object_test.go 2023-01-17 19:56:40 +00:00
hash runtime,hash/maphash: eliminate maphash torture test for -race 2022-12-01 19:24:55 +00:00
html
image
index/suffixarray
internal cmd/go: do not confuse files for standard library packages 2023-01-17 22:30:23 +00:00
io
log
math math/big: fix BitLen performance regression 2022-12-02 17:52:41 +00:00
mime
net net: fix typo in hosts.go 2023-01-06 17:00:29 +00:00
os os: document that Rename is not atomic on non-Unix platforms 2023-01-17 18:12:07 +00:00
path path/filepath: make Join("c:", "/a") return "c:/a" again 2022-12-02 19:48:47 +00:00
plugin
reflect
regexp
runtime runtime: fix performance regression in morestack_noctxt on ppc64 2023-01-16 08:37:36 +00:00
sort
strconv
strings strings: remove redundant symbols 2023-01-17 17:24:17 +00:00
sync sync/atomic: fix the note of atomic.Store 2023-01-09 18:21:12 +00:00
syscall runtime: revert use of __fork to work around Apple atfork bugs 2023-01-10 20:34:19 +00:00
testdata
testing all: fix typos in go file comments 2023-01-09 15:34:31 +00:00
text
time time/tzdata: generate zip constant during cmd/dist 2023-01-17 22:30:53 +00:00
unicode
unsafe
vendor all: update vendored golang.org/x/net 2022-12-07 00:51:44 +00:00
Make.dist
README.vendor README.vendor: minor updates 2023-01-03 20:31:59 +00:00
all.bash
all.bat
all.rc
bootstrap.bash
buildall.bash
clean.bash
clean.bat
clean.rc
cmp.bash
go.mod all: update vendored golang.org/x/net 2022-12-07 00:51:44 +00:00
go.sum all: update vendored golang.org/x/net 2022-12-07 00:51:44 +00:00
make.bash make.bash, make.rc: fix GOROOT detection when GOEXPERIMENT is set 2023-01-17 22:30:56 +00:00
make.bat
make.rc make.bash, make.rc: fix GOROOT detection when GOEXPERIMENT is set 2023-01-17 22:30:56 +00:00
race.bash
race.bat
run.bash run.bash, cmd/dist: document GO_TEST_SHORT and GO_TEST_TIMEOUT_SCALE 2022-12-06 19:52:35 +00:00
run.bat
run.rc

README.vendor

Vendoring in std and cmd
========================

The Go command maintains copies of external packages needed by the
standard library in the src/vendor and src/cmd/vendor directories.

There are two modules, std and cmd, defined in src/go.mod and
src/cmd/go.mod. When a package outside std or cmd is imported
by a package inside std or cmd, the import path is interpreted
as if it had a "vendor/" prefix. For example, within "crypto/tls",
an import of "golang.org/x/crypto/cryptobyte" resolves to
"vendor/golang.org/x/crypto/cryptobyte". When a package with the
same path is imported from a package outside std or cmd, it will
be resolved normally. Consequently, a binary may be built with two
copies of a package at different versions if the package is
imported normally and vendored by the standard library.

Vendored packages are internally renamed with a "vendor/" prefix
to preserve the invariant that all packages have distinct paths.
This is necessary to avoid compiler and linker conflicts. Adding
a "vendor/" prefix also maintains the invariant that standard
library packages begin with a dotless path element.

The module requirements of std and cmd do not influence version
selection in other modules. They are only considered when running
module commands like 'go get' and 'go mod vendor' from a directory
in GOROOT/src.

Maintaining vendor directories
==============================

Before updating vendor directories, ensure that module mode is enabled.
Make sure that GO111MODULE is not set in the environment, or that it is
set to 'on' or 'auto'.

Requirements may be added, updated, and removed with 'go get'.
The vendor directory may be updated with 'go mod vendor'.
A typical sequence might be:

    cd src
    go get golang.org/x/net@latest
    go mod tidy
    go mod vendor

Use caution when passing '-u' to 'go get'. The '-u' flag updates
modules providing all transitively imported packages, not only
the module providing the target package.

Note that 'go mod vendor' only copies packages that are transitively
imported by packages in the current module. If a new package is needed,
it should be imported before running 'go mod vendor'.