Commit Graph

540 Commits

Author SHA1 Message Date
Austin Clements cf2396c70e internal/goexperiment: move baseline configuration to objabi
We need to adjust baseline experiment configuration based on the
configured GOOS and GOARCH, so it can't live in goexperiment. Move it
to objabi.

Change-Id: I65f4ce56902c6c1a82735050773c58f2d1320cc6
Reviewed-on: https://go-review.googlesource.com/c/go/+/310169
Trust: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2021-04-16 03:16:11 +00:00
Lynn Boger 8009a81f7a bytes: add asm implementation for index on ppc64x
This adds an asm implementation of index on ppc64le and
ppc64. It results in a significant improvement in some
of the benchmarks that use bytes.Index.

The implementation is based on a port of the s390x asm
implementation. Comments on the design are found
with the code.

The following improvements occurred on power8:

Index/10       70.7ns ± 0%    18.8ns ± 0%   -73.4
Index/32        165ns ± 0%      95ns ± 0%   -42.6
Index/4K       9.23µs ± 0%    4.91µs ± 0%   -46
Index/4M       9.52ms ± 0%    5.10ms ± 0%   -46.4
Index/64M       155ms ± 0%      85ms ± 0%   -45.1

Count/10       83.0ns ± 0%    32.1ns ± 0%   -61.3
Count/32        178ns ± 0%     109ns ± 0%   -38.8
Count/4K       9.24µs ± 0%    4.93µs ± 0%   -46
Count/4M       9.52ms ± 0%    5.10ms ± 0%   -46.4
Count/64M       155ms ± 0%      85ms ± 0%   -45.1

IndexHard1     2.36ms ± 0%    0.13ms ± 0%   -94.4
IndexHard2     2.36ms ± 0%    1.28ms ± 0%   -45.8
IndexHard3     2.36ms ± 0%    1.19ms ± 0%   -49.4
IndexHard4     2.36ms ± 0%    2.35ms ± 0%    -0.1

CountHard1     2.36ms ± 0%    0.13ms ± 0%   -94.4
CountHard2     2.36ms ± 0%    1.28ms ± 0%   -45.8
CountHard3     2.36ms ± 0%    1.19ms ± 0%   -49.4

IndexPeriodic/IndexPeriodic2  146µs ± 0%       8µs ± 0%   -94
IndexPeriodic/IndexPeriodic4  146µs ± 0%       8µs ± 0%   -94

Change-Id: I7dd2bb7e278726e27f51825ca8b2f8317d460e60
Reviewed-on: https://go-review.googlesource.com/c/go/+/309730
Run-TryBot: Lynn Boger <laboger@linux.vnet.ibm.com>
Reviewed-by: Paul Murphy <murp@ibm.com>
Reviewed-by: Carlos Eduardo Seo <carlos.seo@linaro.org>
Trust: Carlos Eduardo Seo <carlos.seo@linaro.org>
Trust: Lynn Boger <laboger@linux.vnet.ibm.com>
2021-04-15 17:32:04 +00:00
Austin Clements 8f4c5068e0 internal/bytealg: port more performance-critical functions to ABIInternal
CL 308931 ported several runtime assembly functions to ABIInternal so
that compiler-generated ABIInternal calls don't go through ABI
wrappers, but it missed the runtime assembly functions that are
actually defined in internal/bytealg.

This eliminates the cost of wrappers for the BleveQuery and
GopherLuaKNucleotide benchmarks, but there's still more to do for
Tile38.

                                      0-base                1-wrappers
                                     sec/op        sec/op            vs base
BleveQuery                          6.507 ± 0%    6.477 ± 0%  -0.46% (p=0.004 n=20)
GopherLuaKNucleotide                30.39 ± 1%    30.34 ± 0%       ~ (p=0.301 n=20)
Tile38IntersectsCircle100kmRequest 1.038m ± 1%   1.080m ± 2%  +4.03% (p=0.000 n=20)

For #40724.

Change-Id: I0b722443f684fcb997b1d70802c5ed4b8d8f9829
Reviewed-on: https://go-review.googlesource.com/c/go/+/310184
Trust: Austin Clements <austin@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2021-04-15 04:10:33 +00:00
Michael Anthony Knyszek 3eaf75c13a runtime: move next_gc and last_next_gc into gcControllerState
This change moves next_gc and last_next_gc into gcControllerState under
the names heapGoal and lastHeapGoal respectively. These are
fundamentally GC pacer related values, and so it makes sense for them to
live here.

Partially generated by

rf '
    ex . {
	memstats.next_gc -> gcController.heapGoal
	memstats.last_next_gc -> gcController.lastHeapGoal
    }
'

except for updates to comments and gcControllerState methods, where
they're accessed through the receiver, and trace-related renames of
NextGC -> HeapGoal, while we're here.

For #44167.

Change-Id: I1e871ad78a57b01be8d9f71bd662530c84853bed
Reviewed-on: https://go-review.googlesource.com/c/go/+/306603
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
2021-04-14 14:03:30 +00:00
Manlio Perillo cccd3ba912 internal/execabs: replace ioutil.WriteFile with os.WriteFile
Fixes #45532.

Change-Id: I844acd50d6fa1ce918969bbb52f79dd7412d289f
Reviewed-on: https://go-review.googlesource.com/c/go/+/309350
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Tobias Klauser <tobias.klauser@gmail.com>
2021-04-12 21:57:23 +00:00
Ian Lance Taylor 8b859be9c3 internal/poll: ensure that newPoolPipe doesn't return a nil pointer
The function could occasionally return a nil pointer as a non-nil
interface, confusing the calling code.

Fixes #45520

Change-Id: Ifd35613728efa2cee9903177e85d369155074804
Reviewed-on: https://go-review.googlesource.com/c/go/+/309429
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Cuong Manh Le <cuong.manhle.vn@gmail.com>
Reviewed-by: Andy Pan <panjf2000@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
2021-04-12 17:32:47 +00:00
Andy Pan 6382ec1aba internal/poll: fix the intermittent build failures with pipe pool
Correlative CL 308089

Fixes #45059

Change-Id: I1ff9fbf64e6620d651f287ba2a28d40f964d78a3
Reviewed-on: https://go-review.googlesource.com/c/go/+/308329
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
2021-04-10 03:38:08 +00:00
Austin Clements 5159c83641 runtime,cmd/link: include GOEXPERIMENTs in runtime.Version(), "go version X"
This adds the set of GOEXPERIMENTs to the build version if it differs
from the default set of experiments. This exposes the experiment
settings via runtime.Version() and "go version <binary>".

Change-Id: I143dbbc50f66a4cf175469199974e18848075af6
Reviewed-on: https://go-review.googlesource.com/c/go/+/307820
Trust: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2021-04-08 02:17:22 +00:00
Austin Clements a8e55538af cmd/internal/objabi: make GOEXPERIMENT be a diff from default experiments
Right now the rules around handling default-on experiments are
complicated and a bit inconsistent. Notably, objabi.GOEXPERIMENT is
set to a comma-separated list of enabled experiments, but this may not
be the string a user should set the GOEXPERIMENT environment variable
to get that list of experiments: if an experiment is enabled by
default but gets turned off by GOEXPERIMENT, then the string we report
needs to include "no"+experiment to capture that default override.

This complication also seeps into the version string we print for "go
tool compile -V", etc. This logic is further complicated by the fact
that it only wants to include an experiment string if the set of
experiments varies from the default.

This CL rethinks how we handle default-on experiments. Now that
experiment state is all captured in a struct, we can simplify a lot of
this logic. objabi.GOEXPERIMENT will be set based on the delta from
the default set of experiments, which reflects what a user would
actually need to pass on the command line. Likewise, we include this
delta in the "-V" output, which simplifies this logic because if
there's nothing to show in the version string, the delta will be
empty.

Change-Id: I7ed307329541fc2c9f90edd463fbaf8e0cc9e8ee
Reviewed-on: https://go-review.googlesource.com/c/go/+/307819
Trust: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2021-04-08 02:17:20 +00:00
Austin Clements b675e52e95 internal/goexperiment: consolidate experiment-enabled constants
Currently, we have boolean and integral constants for GOEXPERIMENTs in
various places. Consolidate these into automatically generated
constants in the internal/goexperiment package.

Change-Id: I42a49aba2a3b4c722fedea23a613162cd8a67bee
Reviewed-on: https://go-review.googlesource.com/c/go/+/307818
Trust: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2021-04-08 02:17:18 +00:00
Austin Clements 6304b401e4 internal/goexperiment,cmd: consolidate GOEXPERIMENTs into a new package
Currently there's knowledge about the list of GOEXPERIMENTs in a few
different places. This CL introduces a new package and consolidates
the list into one place: the internal/goexperiment.Flags struct type.

This package gives us a central place to document the experiments as
well as the GOEXPERIMENT environment variable itself. It will also
give us a place to put built-time constants derived from the enabled
experiments.

Now the objabi package constructs experiment names by reflecting over
this struct type rather than having a separate list of these names
(this is similar to how the compiler handles command-line flags and
debug options). We also expose a better-typed API to the toolchain for
propagating enabled experiments.

Change-Id: I06e026712b59fe2bd7cd11a869aedb48ffe5a4b7
Reviewed-on: https://go-review.googlesource.com/c/go/+/307817
Trust: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Matthew Dempsky <mdempsky@google.com>
2021-04-08 02:17:16 +00:00
Ian Lance Taylor e985245cd5 net: make ErrClosed and ParseError implement net.Error
Fixes #45357

Change-Id: Iafd41fff232a89be4c88d4b1d66bc3c04d888bcc
Reviewed-on: https://go-review.googlesource.com/c/go/+/307030
Trust: Ian Lance Taylor <iant@golang.org>
Trust: Josh Bleecher Snyder <josharian@gmail.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
2021-04-05 19:15:53 +00:00
KimMachineGun a040ebeb98 all: update references to symbols moved from io/ioutil to io
Update references missed in CL 263142.

For #41190

Change-Id: I778760a6a69bd0440fec0848bdef539c9ccb4ee1
GitHub-Last-Rev: dda42b09ff
GitHub-Pull-Request: golang/go#42874
Reviewed-on: https://go-review.googlesource.com/c/go/+/273946
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Cherry Zhang <cherryyz@google.com>
2021-04-05 17:51:15 +00:00
Austin Clements eaa1ddee84 all: explode GOEXPERIMENT=regabi into 5 sub-experiments
This separates GOEXPERIMENT=regabi into five sub-experiments:
regabiwrappers, regabig, regabireflect, regabidefer, and regabiargs.
Setting GOEXPERIMENT=regabi now implies the working subset of these
(currently, regabiwrappers, regabig, and regabireflect).

This simplifies testing, helps derisk the register ABI project,
and will also help with performance comparisons.

This replaces the -abiwrap flag to the compiler and linker with
the regabiwrappers experiment.

As part of this, regabiargs now enables registers for all calls
in the compiler. Previously, this was statically disabled in
regabiEnabledForAllCompilation, but now that we can control it
independently, this isn't necessary.

For #40724.

Change-Id: I5171e60cda6789031f2ef034cc2e7c5d62459122
Reviewed-on: https://go-review.googlesource.com/c/go/+/302070
Trust: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: David Chase <drchase@google.com>
2021-03-18 16:51:27 +00:00
Tobias Klauser f38b6428a2 crypto/rand, internal/syscall/unix: add support for getentropy syscall on darwin
The getentropy syscall is available on macOS since version 10.12, which
is the minimum required version since Go 1.15.

Change-Id: I294259af0b11df9669e4dc5fa891d2f2f039d91a
Reviewed-on: https://go-review.googlesource.com/c/go/+/302489
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
2021-03-17 22:14:28 +00:00
Andy Pan 15f16706fb internal/poll: eliminate the redundant type conversions of FD.Sysfd
Change-Id: Ib75662f717320510319c696520e645f54eec97f7
Reviewed-on: https://go-review.googlesource.com/c/go/+/301569
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2021-03-16 01:40:41 +00:00
Josh Bleecher Snyder 061a6903a2 all: add internal/itoa package
This replaces five implementations scattered across low level packages.
(And I plan to use it in a sixth soon.)
Three of the five were byte-for-byte identical.

Change-Id: I3bbbeeac63723a487986c912b604e10ad1e042f4
Reviewed-on: https://go-review.googlesource.com/c/go/+/301549
Trust: Josh Bleecher Snyder <josharian@gmail.com>
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
2021-03-14 17:56:50 +00:00
Andy Pan 71330963c0 internal/poll: fix some grammar errors
Change-Id: I25a6424bce9d372fa46e8bdd856095845d3397bf
Reviewed-on: https://go-review.googlesource.com/c/go/+/300889
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
2021-03-12 02:21:04 +00:00
Matthew Dempsky 7fc638d6f1 cmd: move GOEXPERIMENT knob from make.bash to cmd/go
This CL changes GOEXPERIMENT to act like other GO[CONFIG] environment
variables. Namely, that it can be set at make.bash time to provide a
default value used by the toolchain, but then can be manually set when
running either cmd/go or the individual tools (compiler, assembler,
linker).

For example, it's now possible to test rsc.io/tmp/fieldtrack by simply
running:

GOEXPERIMENT=fieldtrack go test -gcflags=-l rsc.io/tmp/fieldtrack \
  -ldflags=-k=rsc.io/tmp/fieldtrack.tracked

without needing to re-run make.bash. (-gcflags=-l is needed because
the compiler's inlining abilities have improved, so calling a function
with a for loop is no longer sufficient to suppress inlining.)

Fixes #42681.

Change-Id: I2cf8995d5d0d05f6785a2ee1d3b54b2cfb3331ca
Reviewed-on: https://go-review.googlesource.com/c/go/+/300991
Trust: Matthew Dempsky <mdempsky@google.com>
Run-TryBot: Matthew Dempsky <mdempsky@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2021-03-11 21:43:04 +00:00
Tobias Klauser 9ece63f064 crypto/rand, internal/syscall/unix: add support for getrandom syscall on solaris
The getrandom syscall is available on Solaris and Illumos, see
https://docs.oracle.com/cd/E88353_01/html/E37841/getrandom-2.html and
https://illumos.org/man/2/getrandom

Change-Id: Id1c65d6a5b2fbc80d20b43d8b32dab137ca950ca
Reviewed-on: https://go-review.googlesource.com/c/go/+/299134
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2021-03-11 08:19:39 +00:00
Tobias Klauser 79e3ee52f4 internal/syscall/unix: unify GetRandom implementation
The implementation of GetRandom for Linux, FreeBSD and DragonflyBSD can
be shared.

Also remove GRND_INSECURE on DragonflyBSD as pointed out by Ian in the
review of CL 269999.

Change-Id: I5bf4c1bd51ddb2ad600652a57e0bc1bafa1cf40d
Reviewed-on: https://go-review.googlesource.com/c/go/+/299133
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2021-03-11 08:19:31 +00:00
David Carlier cf59850466 crypto/rand: supports for getrandom syscall in DragonFlyBSD
Since the 5.7 release, DragonFlyBSD supports as well
the getrandom function, the actual stable is 5.8.

Change-Id: I2b8fc468771b10ac12b38ea7e8e5314342de6375
GitHub-Last-Rev: c5c496f418
GitHub-Pull-Request: golang/go#42617
Reviewed-on: https://go-review.googlesource.com/c/go/+/269999
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Trust: Ian Lance Taylor <iant@golang.org>
Trust: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
2021-03-10 09:01:05 +00:00
Andy Pan 643d240a11 internal/poll: implement a pipe pool for splice() call
In scenarios where splice() is called, splice() is usually called not just once, but many times,
which means that a lot of pipes will be created and destroyed frequently, costing an amount of system resources
and slowing down performance, thus I suggest that we add a pipe pool for reusing pipes.

Benchmark tests:

goos: linux
goarch: amd64
pkg: internal/poll
cpu: AMD EPYC 7K62 48-Core Processor

name                  old time/op    new time/op    delta
SplicePipe-8            1.36µs ± 1%    0.02µs ± 0%   -98.57%  (p=0.001 n=7+7)
SplicePipeParallel-8     747ns ± 4%       4ns ± 0%   -99.41%  (p=0.001 n=7+7)

name                  old alloc/op   new alloc/op   delta
SplicePipe-8             24.0B ± 0%      0.0B       -100.00%  (p=0.001 n=7+7)
SplicePipeParallel-8     24.0B ± 0%      0.0B       -100.00%  (p=0.001 n=7+7)

name                  old allocs/op  new allocs/op  delta
SplicePipe-8              1.00 ± 0%      0.00       -100.00%  (p=0.001 n=7+7)
SplicePipeParallel-8      1.00 ± 0%      0.00       -100.00%  (p=0.001 n=7+7)

Fixes #42740

Change-Id: Idff654b7264342084e089b5ba796c87c380c471b
Reviewed-on: https://go-review.googlesource.com/c/go/+/271537
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Brad Fitzpatrick <bradfitz@golang.org>
2021-03-10 03:52:48 +00:00
Michael Anthony Knyszek c5a1c2276e reflect: use global variables for register count
This change switches reflect to use global variables for ABI-related
register counts instead of internal/abi constants. The advantage of
doing so is that we can make the internal/abi constants non-zero and
enable the runtime register argument spiller/unspiller even if they're
not used. It's basically turning two things we need to flip when we
switch to the register ABI into one.

It also paves the way for testing the reflect register ABI path
independently, because now we can switch the global variables at will
and run the register-assignment algorithm in tests without having the
rest of the runtime be broken.

Change-Id: Ie23629a37a5c80aeb24909d4bd9eacbd3f0c06d9
Reviewed-on: https://go-review.googlesource.com/c/go/+/293149
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2021-03-05 15:34:15 +00:00
Tobias Klauser ebb92dfed9 internal/poll, runtime: handle netpollopen error in poll_runtime_pollOpen
When netpollopen in poll_runtime_pollOpen returns an error, the work in
runtime_pollUnblock and runtime_pollClose can be avoided since the
underlying system call to set up the poller failed.

E.g. on linux, this avoids calling netpollclose and thus epoll_ctl(fd,
EPOLL_CTL_DEL, ...) in case the file does not support epoll, i.e.
epoll_ctl(fd, EPOLL_CTL_ADD, ...) in netpollopen failed.

Fixes #44552

Change-Id: I564d90340fd1ab3a6490526353616a447ae0cfb8
Reviewed-on: https://go-review.googlesource.com/c/go/+/297392
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2021-03-02 06:08:56 +00:00
John Bampton 6ba4a300d8 docs: fix spelling
Change-Id: Ib689e5793d9cb372e759c4f34af71f004010c822
GitHub-Last-Rev: d63798388e
GitHub-Pull-Request: golang/go#44259
Reviewed-on: https://go-review.googlesource.com/c/go/+/291949
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Matthew Dempsky <mdempsky@google.com>
Trust: Robert Griesemer <gri@golang.org>
2021-02-24 04:11:43 +00:00
Andy Pan a78b0e6721 internal/poll: fix the verbose condition in splice
Change-Id: I0b433ea1a78632de20ea58c48c9be0f1fb6eb083
Reviewed-on: https://go-review.googlesource.com/c/go/+/271499
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
2021-02-23 10:04:13 +00:00
Russ Cox d4b2638234 all: go fmt std cmd (but revert vendor)
Make all our package sources use Go 1.17 gofmt format
(adding //go:build lines).

Part of //go:build change (#41184).
See https://golang.org/design/draft-gobuild

Change-Id: Ia0534360e4957e58cd9a18429c39d0e32a6addb4
Reviewed-on: https://go-review.googlesource.com/c/go/+/294430
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2021-02-20 03:54:50 +00:00
Cherry Zhang 03cea563d1 [dev.regabi] all: merge master (5faf941) into dev.regabi
Merge List:

+ 2021-02-16 5faf941df0 internal/goversion: update Version to 1.17
+ 2021-02-16 6f3da9d2f6 README: pull gopher image from website
+ 2021-02-16 098504c73f cmd/link: generate trampoline for inter-dependent packages
+ 2021-02-16 1004a7cb31 runtime/metrics: update documentation to current interface
+ 2021-02-16 6530f2617f doc/go1.16: remove draft notice
+ 2021-02-16 353e111455 doc/go1.16: fix mismatched id attribute
+ 2021-02-16 f0d23c9dbb internal/poll: netpollcheckerr before sendfile
+ 2021-02-16 0cb3415154 doc: remove all docs not tied to distribution
+ 2021-02-16 626ef08127 doc: remove install.html and install-source.html
+ 2021-02-16 30641e36aa internal/poll: if copy_file_range returns 0, assume it failed
+ 2021-02-15 33d72fd412 doc/faq: update generics entry to reflect accepted proposal
+ 2021-02-15 852ce7c212 cmd/go: provide a more helpful suggestion for "go vet -?"
+ 2021-02-13 66c27093d0 cmd/link: fix typo in link_test.go

Change-Id: I98f047b79b93c5ceb344dd43408bcb919b23aeb3
2021-02-16 16:18:19 -05:00
Bryan C. Mills 5faf941df0 internal/goversion: update Version to 1.17
(The corresponding update for the last release cycle was CL 248038.)

For #40705.

Change-Id: I13becdc4c3718a1c6986876ec56879cce3bcb34f
Reviewed-on: https://go-review.googlesource.com/c/go/+/275297
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2021-02-16 21:07:42 +00:00
Michael Anthony Knyszek 8cfbf34dd9 internal/abi: set register count constants to zero for regabi experiment
This change sets the register count constants to zero for the
GOEXPERIMENT regabi because currently the users of it (i.e. reflect)
will be broken, since they expect Go functions that implement the new
ABI.

Change-Id: Id3e874c61821a36605eb4e1cccdee36a2759f303
Reviewed-on: https://go-review.googlesource.com/c/go/+/292649
Reviewed-by: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
2021-02-16 19:40:58 +00:00
Michael Anthony Knyszek e0215315f5 [dev.regabi] reflect: support for register ABI on amd64 for reflect.(Value).Call
This change adds support for the new register ABI on amd64 to
reflect.(Value).Call. If internal/abi's register counts are non-zero,
reflect will try to set up arguments in registers on the Call path.

Note that because the register ABI becomes ABI0 with zero registers
available, this should keep working as it did before.

This change does not add any tests for the register ABI case because
there's no way to do so at the moment.

For #40724.

Change-Id: I8aa089a5aa5a31b72e56b3d9388dd3f82203985b
Reviewed-on: https://go-review.googlesource.com/c/go/+/272568
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
2021-02-16 15:51:50 +00:00
Wei Fu f0d23c9dbb internal/poll: netpollcheckerr before sendfile
In net/http package, the ServeContent/ServeFile doesn't check the I/O
timeout error from chunkWriter or *net.TCPConn, which means that both
HTTP status and headers might be missing when WriteTimeout happens. If
the poll.SendFile() doesn't check the *poll.FD state before sending
data, the client will only receive the response body with status and
report "malformed http response/status code".

This patch is to enable netpollcheckerr before sendfile, which should
align with normal *poll.FD.Write() and Splice().

Fixes #43822

Change-Id: I32517e3f261bab883a58b577b813ef189214b954
Reviewed-on: https://go-review.googlesource.com/c/go/+/285914
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com>
2021-02-16 11:01:41 +00:00
Ian Lance Taylor 30641e36aa internal/poll: if copy_file_range returns 0, assume it failed
On current Linux kernels copy_file_range does not correctly handle
files in certain special file systems, such as /proc. For those file
systems it fails to copy any data and returns zero. This breaks Go's
io.Copy for those files.

Fix the problem by assuming that if copy_file_range returns 0 the
first time it is called on a file, that that file is not supported.
In that case fall back to just using read. This will force an extra
system call when using io.Copy to copy a zero-sized normal file,
but at least it will work correctly.

For #36817
Fixes #44272

Change-Id: I02e81872cb70fda0ce5485e2ea712f219132e614
Reviewed-on: https://go-review.googlesource.com/c/go/+/291989
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
2021-02-16 00:46:04 +00:00
Michael Anthony Knyszek baa6c75dce [dev.regabi] internal/abi: add new internal/abi package for ABI constants
This change creates a new internal std package internal/abi which is
intended to hold constants with platform-specific values related to
our ABI that is useful to different std packages, such as runtime and
reflect.

For #40724.

Change-Id: Ie7ae7f687629cd3d613ba603e9371f0887601fe6
Reviewed-on: https://go-review.googlesource.com/c/go/+/272567
Trust: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Austin Clements <austin@google.com>
2021-02-12 22:11:52 +00:00
Roland Shoemaker 953d1feca9 all: introduce and use internal/execabs
Introduces a wrapper around os/exec, internal/execabs, for use in
all commands. This wrapper prevents exec.LookPath and exec.Command from
running executables in the current directory.

All imports of os/exec in non-test files in cmd/ are replaced with
imports of internal/execabs.

This issue was reported by RyotaK.

Fixes CVE-2021-3115
Fixes #43783

Change-Id: I0423451a6e27ec1e1d6f3fe929ab1ef69145c08f
Reviewed-on: https://team-review.git.corp.google.com/c/golang/go-private/+/955304
Reviewed-by: Russ Cox <rsc@google.com>
Reviewed-by: Katie Hockman <katiehockman@google.com>
Reviewed-on: https://go-review.googlesource.com/c/go/+/284783
Run-TryBot: Roland Shoemaker <roland@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Roland Shoemaker <roland@golang.org>
2021-01-21 19:10:18 +00:00
Jason A. Donenfeld bb5075a525 syscall: remove RtlGenRandom and move it into internal/syscall
There's on need to expose this to the frozen syscall package, and it
also doesn't need to be unsafe. So we move it into internal/syscall and
have the generator make a safer function signature.

Fixes #43704.

Change-Id: Iccae69dc273a0aa97ee6846eb537f1dc1412f2de
Reviewed-on: https://go-review.googlesource.com/c/go/+/283992
Run-TryBot: Jason A. Donenfeld <Jason@zx2c4.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Trust: Jason A. Donenfeld <Jason@zx2c4.com>
2021-01-15 18:42:27 +00:00
Russ Cox f1980efb92 all: update to use os.ReadDir where appropriate
os.ReadDir is a replacement for ioutil.ReadDir that returns
a slice of fs.DirEntry instead of fs.FileInfo, meaning it is the
more efficient form.

This CL updates call sites throughout the Go source tree
wherever possible. As usual, code built using the Go 1.4
bootstrap toolchain is not included. There is also a use in
go/build that appears in the public API and can't be changed,
at least not without additional changes.

Fixes #42026.

Change-Id: Icfc9dd52c6045020f6830e22c72128499462d561
Reviewed-on: https://go-review.googlesource.com/c/go/+/266366
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-12-09 19:12:27 +00:00
Russ Cox 4f1b0a44cb all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp
As part of #42026, these helpers from io/ioutil were moved to os.
(ioutil.TempFile and TempDir became os.CreateTemp and MkdirTemp.)

Update the Go tree to use the preferred names.

As usual, code compiled with the Go 1.4 bootstrap toolchain
and code vendored from other sources is excluded.

ReadDir changes are in a separate CL, because they are not a
simple search and replace.

For #42026.

Change-Id: If318df0216d57e95ea0c4093b89f65e5b0ababb3
Reviewed-on: https://go-review.googlesource.com/c/go/+/266365
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-12-09 19:12:23 +00:00
Martin Möhrmann c155931974 internal/cpu: add darwin/arm64 CPU feature detection support
Fixes #42747

Change-Id: I6b1679348c77161f075f0678818bb003fc0e8c86
Reviewed-on: https://go-review.googlesource.com/c/go/+/271989
Trust: Martin Möhrmann <moehrmann@google.com>
Run-TryBot: Martin Möhrmann <martisch@uos.de>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
2020-12-07 07:59:54 +00:00
Ikko Ashimine 3b2a578166 internal/cpu: fix typo in cpu_arm64.go
auxillary -> auxiliary

Change-Id: I7c29c4a63d236c3688b8e4f5af70650d43cd89c0
GitHub-Last-Rev: d4a18c71a1
GitHub-Pull-Request: golang/go#43024
Reviewed-on: https://go-review.googlesource.com/c/go/+/275592
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Keith Randall <khr@golang.org>
2020-12-05 16:20:01 +00:00
Martin Möhrmann dda2991c2e internal/cpu: disable FMA when OSXSAVE is not enabled on x86
All instructions in the FMA extension on x86 are VEX prefixed.
VEX prefixed instructions generally require OSXSAVE to be enabled.

The execution of FMA instructions emitted by the Go compiler on amd64
will generate an invalid opcode exception if OSXSAVE is not enabled.

Fixes #41022

Change-Id: I49881630e7195c804110a2bd81b5bec8cac31ba8
Reviewed-on: https://go-review.googlesource.com/c/go/+/274479
Trust: Martin Möhrmann <moehrmann@google.com>
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
2020-12-03 16:31:53 +00:00
Ian Lance Taylor 66c0264506 net, internal/poll: reset value before adding in minor kernel version
Fixes #42733

Change-Id: I5446aeb5de13cd70212755fb12c9bc484f343c74
Reviewed-on: https://go-review.googlesource.com/c/go/+/271846
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Cuong Manh Le <cuong.manhle.vn@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
2020-11-20 04:21:00 +00:00
Hanlin Shi 0bb6115dd6 internal/fmtsort: sort the unsafe pointers in map
Currently storing keys that contain unsafe.
Pointer in a map could result inruntime panic when printing the map.
The root cause is that unsafe.Pointer is not comparable.

Fixes #42622.

Change-Id: Ie3bae7ee4945041843b66514de6227212a3da73e
GitHub-Last-Rev: d12d41302e
GitHub-Pull-Request: golang/go#42623
Reviewed-on: https://go-review.googlesource.com/c/go/+/270277
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-11-19 02:17:10 +00:00
Tobias Klauser 1c7650aa93 internal/poll: use copy_file_range only on Linux kernel >= 5.3
https://man7.org/linux/man-pages/man2/copy_file_range.2.html#VERSIONS states:

  A major rework of the kernel implementation occurred in 5.3.  Areas
  of the API that weren't clearly defined were clarified and the API
  bounds are much more strictly checked than on earlier kernels.
  Applications should target the behaviour and requirements of 5.3
  kernels.

Rather than attempting to detect the file system for source and
destination files (which means two additional statfs syscalls) and skip
copy_file_range in case of known defects (e.g. CIFS -> CIFS), just
assume copy_file_range to be broken on kernels < 5.3.

Fixes #42400

Change-Id: I3a531296182c1d6e341772cc9d2be5bf83e52575
Reviewed-on: https://go-review.googlesource.com/c/go/+/268338
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-11-10 09:01:41 +00:00
Russ Cox 979e137609 cmd/go: add GOVCS setting to control version control usage
The go command runs commands like git and hg to download modules.
In the past, we have had problems with security bugs in version
control systems becoming security bugs in “go get”.

The original modules draft design removed use of these commands
entirely, saying:

> We want to move away from invoking version control tools such as bzr,
> fossil, git, hg, and svn to download source code. These fragment the
> ecosystem: packages developed using Bazaar or Fossil, for example, are
> effectively unavailable to users who cannot or choose not to install
> these tools. The version control tools have also been a source of
> exciting security problems. It would be good to move them outside the
> security perimeter.

The removal of these commands was not possible in the end: being able
to fetch directly from Git repos is too important, especially for
closed source. But the security exposure has not gone away.
We remain vulnerable to problems in VCS systems, especially the less
scrutinized ones.

This change adds a GOVCS setting to let users control which version
control systems are allowed by default.

It also changes the default allowed version control systems to git and hg
for public code and any version control system for private code
(import path or module path matched by the GOPRIVATE setting).

See the changes in alldocs.go for detailed documentation.
See #41730 for proposal and discussion.

Fixes #41730.

[Replay of CL 266420. See changes from Patch Set 1 for updates to fix
a few long tests.]

Change-Id: I4fe93804548956c42aea985368b4571bdb220f48
Reviewed-on: https://go-review.googlesource.com/c/go/+/267888
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
2020-11-09 15:46:56 +00:00
Dmitri Shuralyov 34c09695db cmd/go: revert "add GOVCS setting to control version control usage"
This reverts CL 266420.

Reason for revert: tests aren't passing on linux-{386,amd64}-longtest.

Change-Id: Icec47cded795a51ef7569dfb2d93d9211b4fb578
Reviewed-on: https://go-review.googlesource.com/c/go/+/267799
Trust: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2020-11-05 15:16:57 +00:00
Martin Möhrmann 3510a1e32c internal/cpu: fix and cleanup ARM64 cpu feature fields and options
Remove all cpu features from the ARM64 struct that are not initialized
to reduce cache lines used and to avoid those features being
accidentially used without actual detection if they are present.

Add missing option to mask the CPUID feature.

Change-Id: I94bf90c0655de1af2218ac72117ac6c52adfc289
Reviewed-on: https://go-review.googlesource.com/c/go/+/267658
Run-TryBot: Martin Möhrmann <moehrmann@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Trust: Martin Möhrmann <moehrmann@google.com>
2020-11-05 10:46:08 +00:00
Russ Cox c018eec1f3 cmd/go: add GOVCS setting to control version control usage
The go command runs commands like git and hg to download modules.
In the past, we have had problems with security bugs in version
control systems becoming security bugs in “go get”.

The original modules draft design removed use of these commands
entirely, saying:

> We want to move away from invoking version control tools such as bzr,
> fossil, git, hg, and svn to download source code. These fragment the
> ecosystem: packages developed using Bazaar or Fossil, for example, are
> effectively unavailable to users who cannot or choose not to install
> these tools. The version control tools have also been a source of
> exciting security problems. It would be good to move them outside the
> security perimeter.

The removal of these commands was not possible in the end: being able
to fetch directly from Git repos is too important, especially for
closed source. But the security exposure has not gone away.
We remain vulnerable to problems in VCS systems, especially the less
scrutinized ones.

This change adds a GOVCS setting to let users control which version
control systems are allowed by default.

It also changes the default allowed version control systems to git and hg
for public code and any version control system for private code
(import path or module path matched by the GOPRIVATE setting).

See the changes in alldocs.go for detailed documentation.
See #41730 for proposal and discussion.

Fixes #41730.

Change-Id: I1999ddf7445b36a7572965be5897c7a1ff7f4265
Reviewed-on: https://go-review.googlesource.com/c/go/+/266420
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2020-11-05 00:21:39 +00:00
Tobias Klauser 633f9e2060 internal/poll: treat copy_file_range EIO as not-handled
Fixes #42334

Change-Id: Ife51df4e7d2539a04393abfdec45e3f902975fca
Reviewed-on: https://go-review.googlesource.com/c/go/+/266940
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2020-11-04 06:42:33 +00:00