go/src
Dmitri Shuralyov ecf4ebf100 cmd/internal/moddeps: check content of all modules in GOROOT
Expand the scope of the TestAllDependenciesVendored test to check
that all modules in GOROOT are tidy, that packages are vendored,
the vendor content matches the upstream copy exactly, and that
bundled packages are re-generated (using x/tools/cmd/bundle at
the version selected in cmd module; this is deterministic and
guaranteed to be updated over time).

This is done in a conceptually simple way:

1.	Make a temporary copy of the entire GOROOT tree (except .git),
	one that is safe to modify.
2.	Run a list of high-level commands, the same commands we expect
	Go developers should be able to run in a normal complete GOROOT
	tree to make it clean and tidy.
3.	Diff the end result with the original GOROOT tree being tested
	to catch any unexpected differences.

The current set of commands that are run require the cmd/go command,
and a functional compiler itself (because re-generating the syscall
package involves a directive like //go:generate go run [...]). As a
result, copying a large majority of the GOROOT tree is a requirement.

Instead of looking for the few files or directories that can we can
get away not copying (e.g., the testdata directories aren't strictly
needed at this time), we opt not to optimize and just do the simple
copy. This is motivated by these reasons:

•	We end up having a complete, normal GOROOT tree, one that happens
	to be located at another path. There's a very high likelihood that
	module management/code generation commands, both the ones we run
	today and any additional ones that we might want to add in the
	future, will result in correct results even as the Go project
	evolves over time.

•	Having a completely stand-alone copy of the GOROOT tree without
	symlinks minimizes the risk of some of the module management/code
	generation commands, either now or in the future, from modifying
	the user's original GOROOT tree, something that should not happen
	during test execution. Overlays achieved with symlinks work well
	when we can guarantee only new files are added, but that isn't
	the case here.

•	Copying the entire GOROOT (without .git), takes around 5 seconds
	on a fairly modern computer with an SSD. The most we can save is
	a couple of seconds.

(We make some minor exceptions: the GOROOT/.git directory isn't copied,
and GOROOT/{bin,pkg} are deemed safe to share and thus symlink instead
of copying. If these optimizations cease to be viable to make, we'll
need to remove them.)

Since this functionality is fairly expensive to execute and requires
network access, it runs only when the test is executed without -short
flag. The previous behavior of the TestAllDependenciesVendored test is
kept in -short test mode. all.bash runs package tests with -short flag,
so its behavior is unchanged. The expectation is that the new test will
run on some of the longtest builders to catch problems. Users can invoke
the test manually 'go test cmd/internal/moddeps' (and it's run as part
of 'go test cmd', again, only when -short flag isn't provided).

On a 2017 MacBook Pro, a successful long test takes under 15 seconds,
which should be within scope of all long tests that are selected by
'go test std cmd'. We may further adjust when and where the test runs
by default based on our experience.

Fixes #36852.
Fixes #41409.
Fixes #43687.
Updates #43440.

Change-Id: I9eb85205fec7ec62e3f867831a0a82e3c767f618
Reviewed-on: https://go-review.googlesource.com/c/go/+/283643
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
2021-01-20 22:16:54 +00:00
..
archive archive/tar: fix typo in comment 2021-01-08 02:03:24 +00:00
bufio bufio: make string(int) conversion safer 2020-11-28 03:50:50 +00:00
builtin
bytes bytes: add example for (*Buffer).Bytes 2020-11-11 20:51:00 +00:00
cmd cmd/internal/moddeps: check content of all modules in GOROOT 2021-01-20 22:16:54 +00:00
compress all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp 2020-12-09 19:12:23 +00:00
container
context
crypto syscall: remove RtlGenRandom and move it into internal/syscall 2021-01-15 18:42:27 +00:00
database/sql database/sql: fix typo in comment 2020-12-10 18:23:15 +00:00
debug debug/dwarf: don't try to parse addr/rnglists header 2020-12-14 18:06:06 +00:00
embed embed: treat uninitialized FS as empty 2021-01-19 16:31:09 +00:00
encoding encoding/asn1: document unmarshaling behavior for IMPLICIT string fields 2021-01-12 18:32:48 +00:00
errors
expvar
flag
fmt
go cmd/compile: require 'go 1.16' go.mod line for //go:embed 2021-01-19 20:07:52 +00:00
hash all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp 2020-12-09 19:12:23 +00:00
html html/template: attach functions to namespace 2021-01-07 17:51:29 +00:00
image all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp 2020-12-09 19:12:23 +00:00
index/suffixarray all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp 2020-12-09 19:12:23 +00:00
internal syscall: remove RtlGenRandom and move it into internal/syscall 2021-01-15 18:42:27 +00:00
io io/fs: minor corrections to Glob doc 2021-01-08 15:41:36 +00:00
log all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp 2020-12-09 19:12:23 +00:00
math math/rand: fix typo in comment 2021-01-04 17:59:30 +00:00
mime all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp 2020-12-09 19:12:23 +00:00
net net/http/httputil: fix deadlock in DumpRequestOut 2021-01-06 15:02:50 +00:00
os os: do not close syscall.Stdin in TestReadStdin 2021-01-18 17:21:53 +00:00
path all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp 2020-12-09 19:12:23 +00:00
plugin
reflect Revert "reflect: support multiple keys in struct tags" 2021-01-07 23:43:25 +00:00
regexp regexp/syntax: add note about Unicode character classes 2020-11-25 15:10:09 +00:00
runtime runtime: don't adjust timer pp field in timerWaiting status 2021-01-20 19:55:44 +00:00
sort all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp 2020-12-09 19:12:23 +00:00
strconv all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp 2020-12-09 19:12:23 +00:00
strings
sync sync: use 386 instead of x86-32 to refer to the 32 bit x86 architecture 2020-11-23 05:57:35 +00:00
syscall syscall: remove RtlGenRandom and move it into internal/syscall 2021-01-15 18:42:27 +00:00
testdata
testing testing/fstest,os: clarify racy behavior of TestFS 2021-01-08 16:34:00 +00:00
text text/template: error on range over send channel 2020-12-15 13:59:00 +00:00
time lib/time, time/tzdata: update tzdata to 2020f 2020-12-30 22:51:30 +00:00
unicode
unsafe
vendor std: update golang.org/x/net to 20201209123823-ac852fbbde11 2020-12-10 18:27:38 +00:00
Make.dist
README.vendor
all.bash
all.bat
all.rc
bootstrap.bash
buildall.bash
clean.bash
clean.bat
clean.rc
cmp.bash
go.mod std: update golang.org/x/net to 20201209123823-ac852fbbde11 2020-12-10 18:27:38 +00:00
go.sum std: update golang.org/x/net to 20201209123823-ac852fbbde11 2020-12-10 18:27:38 +00:00
make.bash
make.bat
make.rc
race.bash
race.bat
run.bash
run.bat
run.rc

README.vendor

Vendoring in std and cmd
========================

The Go command maintains copies of external packages needed by the
standard library in the src/vendor and src/cmd/vendor directories.

In GOPATH mode, imports of vendored packages are resolved to these
directories following normal vendor directory logic
(see golang.org/s/go15vendor).

In module mode, std and cmd are modules (defined in src/go.mod and
src/cmd/go.mod). When a package outside std or cmd is imported
by a package inside std or cmd, the import path is interpreted
as if it had a "vendor/" prefix. For example, within "crypto/tls",
an import of "golang.org/x/crypto/cryptobyte" resolves to
"vendor/golang.org/x/crypto/cryptobyte". When a package with the
same path is imported from a package outside std or cmd, it will
be resolved normally. Consequently, a binary may be built with two
copies of a package at different versions if the package is
imported normally and vendored by the standard library.

Vendored packages are internally renamed with a "vendor/" prefix
to preserve the invariant that all packages have distinct paths.
This is necessary to avoid compiler and linker conflicts. Adding
a "vendor/" prefix also maintains the invariant that standard
library packages begin with a dotless path element.

The module requirements of std and cmd do not influence version
selection in other modules. They are only considered when running
module commands like 'go get' and 'go mod vendor' from a directory
in GOROOT/src.

Maintaining vendor directories
==============================

Before updating vendor directories, ensure that module mode is enabled.
Make sure GO111MODULE=off is not set ('on' or 'auto' should work).

Requirements may be added, updated, and removed with 'go get'.
The vendor directory may be updated with 'go mod vendor'.
A typical sequence might be:

    cd src
    go get -d golang.org/x/net@latest
    go mod tidy
    go mod vendor

Use caution when passing '-u' to 'go get'. The '-u' flag updates
modules providing all transitively imported packages, not only
the module providing the target package.

Note that 'go mod vendor' only copies packages that are transitively
imported by packages in the current module. If a new package is needed,
it should be imported before running 'go mod vendor'.