Commit Graph

82 Commits

Author SHA1 Message Date
suntala 9e026bf9cc testing: add Output
Output is a method on T, B and F. It provides an io.Writer that writes
to the same test output stream as TB.Log. The new output writer is
used to refactor the implementation of Log. It maintains the formatting
provided by Log while making call site information optional.
Additionally, it provides buffering of log messages. This fixes and
expands on
https://go-review.googlesource.com/c/go/+/646956.

For #59928.

Change-Id: I08179c35a681f601cf125c0f4aeb648bc10c7a9f
GitHub-Last-Rev: e6e202793c
GitHub-Pull-Request: golang/go#73703
Reviewed-on: https://go-review.googlesource.com/c/go/+/672395
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Auto-Submit: Alan Donovan <adonovan@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Auto-Submit: Jonathan Amsterdam <jba@google.com>
2025-05-13 15:43:42 -07:00
Jonathan Amsterdam afd6b0bc66 Revert "testing: add Output"
This reverts commit 8d189f188e.

Reason for revert: failing test

Change-Id: I951087eaef7818697acf87e3206003bcc8a81ee2
Reviewed-on: https://go-review.googlesource.com/c/go/+/672335
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Auto-Submit: Michael Knyszek <mknyszek@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
2025-05-13 12:26:50 -07:00
suntala 8d189f188e testing: add Output
Output is a method on T, B and F. It provides an io.Writer that writes
to the same test output stream as TB.Log. The new output writer is
used to refactor the implementation of Log. It maintains the formatting
provided by Log while making call site information optional.
Additionally, it provides buffering of log messages.

Co-authored-by: Aleks Fazlieva <britishrum@users.noreply.github.com>

Fixes #59928.

Change-Id: I29090b3d4f61f7334388b373ec18750d5637aafa
GitHub-Last-Rev: 18af0e1526
GitHub-Pull-Request: golang/go#71575
Reviewed-on: https://go-review.googlesource.com/c/go/+/646956
Reviewed-by: Arati <artichaut2023@gmail.com>
Auto-Submit: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Jonathan Amsterdam <jba@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
2025-05-13 08:06:08 -07:00
Sean Liao 8270b858ee testing: use more doc links
Change-Id: Ide372735165b7510fd8d7588451a37fa743e59c1
Reviewed-on: https://go-review.googlesource.com/c/go/+/668915
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Commit-Queue: Alan Donovan <adonovan@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
Auto-Submit: Alan Donovan <adonovan@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
2025-04-30 08:21:07 -07:00
Ian Lance Taylor 971448ddf8 testing: support B.Context and F.Context
CL 603959 added T.Context for #36532.

The discussion on the proposal only mentions t.Context.
However, the implementation of CL 603959 also added B.Context and F.Context.
They were added to the API listing, and B.Context was mentioned in
the release notes.

Unfortunately, the new B.Context and F.Context methods always
returned nil, rather than a context.Context value.

This change adds a working implementation of B.Context and F.Context.

For #36532
Fixes #70866

Change-Id: I8a44e6649fb658e4f641ffb7efd08b4374f578ef
Reviewed-on: https://go-review.googlesource.com/c/go/+/637236
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: David Chase <drchase@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
2024-12-18 08:18:04 -08:00
Vladimir Varankin 440c9ee73d testing: rename testContext to testState
Following up to CL 603959, update internals of testing package to
reduce the confusion around "context". The changes rename
testContext/benchContext/fuzzContext to testState/benchState/fuzzState.

Change-Id: Ib8855dab456d41ab343488fcf5fefff2431f7b72
Reviewed-on: https://go-review.googlesource.com/c/go/+/607555
Auto-Submit: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Alan Donovan <adonovan@google.com>
Reviewed-by: Damien Neil <dneil@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-08-21 20:52:09 +00:00
Egon Elbre 1c4704991a testing: use QueryPerformanceCounter on Windows
Windows time.Now granularity is around 0.5ms on modern systems,
which introduces a significant noise into benchmark results.
Instead of relying time.Now use QueryPerformanceCounter, which
has significantly better granularity compared to time.Now.

 │ TimeNow-32  │        HighPrecisionTimeNow-32        │
 │   sec/op    │    sec/op     vs base                 │
   4.812n ± 0%   30.580n ± 0%  +535.43% (p=0.000 n=20)

Fixes #31160

Change-Id: Ib2a574d638c9c6762a2524212def02265574e267
Reviewed-on: https://go-review.googlesource.com/c/go/+/557315
Reviewed-by: Quim Muntal <quimmuntal@gmail.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
2024-04-26 22:55:25 +00:00
guoguangwu 5d4c6088be testing: fix typo in comment
Change-Id: I5aa6093b0199df1ef5b0ad0fcfa651a4b990bfd5
GitHub-Last-Rev: b053d993eb
GitHub-Pull-Request: golang/go#65986
Reviewed-on: https://go-review.googlesource.com/c/go/+/567655
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Auto-Submit: Carlos Amedee <carlos@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2024-02-28 20:32:54 +00:00
Lasse Folger 6317fb41e9 Revert "testing: simplify concurrency and cleanup logic"
reverts CL 506755

Reason for revert: This leads to deadlocks of tests in Google internal testing

For #64402

Change-Id: I78329fc9dcc2377e7e880b264ac1d18d577ef99a
Reviewed-on: https://go-review.googlesource.com/c/go/+/544895
Auto-Submit: Bryan Mills <bcmills@google.com>
Auto-Submit: Lasse Folger <lassefolger@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
2023-11-27 16:49:24 +00:00
Bryan C. Mills e6b76bfc46 testing: simplify concurrency and cleanup logic
While investigating #60083, I found a couple of bugs (notably #61034)
that had slipped through code review in part because the concurrency
patterns used in the testing package were too complex for me to fully
reason about. This change adjusts those patterns to be more in line
with current idioms, and to reduce the number of special cases that
depend on details that should be orthogonal. (For example: the details
of how we invoke the Cleanup functions should not depend on whether
the test happened to run any parallel subtests.)

In the process, this change fixes a handful of bugs:

- Concurrent calls to Run (explicitly allowed by TestParallelSub)
  could previously drive the testcontext.running count negative,
  causing the number of running parallel tests to exceed the -parallel
  flag.

- The -failfast flag now takes effect immediately on failure. It no
  longer delays until the test finishes, and no longer misses failures
  during cleanup (fixing #61034).

- If a Cleanup function calls runtime.Goexit (typically via t.FailNow)
  during a panic, Cleanup functions from its parent tests are no
  longer skipped and buffered logs from its parent tests are now
  flushed.

- The time reported for a test with subtests now includes the time spent
  running those subtests, regardless of whether they are parallel.
  (Previously, non-parallel subtests were included but parallel subtests
  were not.)

- Calls to (*B).Run in iterations after the first are now diagnosed
  with a panic. (This diagnoses badly-behaved benchmarks: if Run is
  called during the first iteration, no subsequent iterations are
  supposed to occur.)

Fixes #61034.

Change-Id: I3797f6ef5210a3d2d5d6c2710d3f35c0219b02ea
Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-linux-amd64-longtest-race,gotip-windows-amd64-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/506755
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2023-11-21 22:58:46 +00:00
cui fliter 0889b39ff1 testing: add available godoc link
Change-Id: I8f4d097601796f53176d490cddf8832b7caa4c05
Reviewed-on: https://go-review.googlesource.com/c/go/+/539836
Run-TryBot: shuang cui <imcusg@gmail.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2023-11-08 17:55:47 +00:00
Bryan C. Mills 55b8e16b2e testing: use monotonic counts to attribute races in subtests
This implements the approach I described in
https://go-review.git.corp.google.com/c/go/+/494057/1#message-5c9773bded2f89b4058848cb036b860aa6716de3.

Specifically:

- Each level of test atomically records the cumulative number of races
  seen as of the last race-induced test failure.

- When a subtest fails, it logs the race error, and then updates its
  parents' counters so that they will not log the same error.

- We check each test or benchmark for races before it starts running
  each of its subtests or sub-benchmark, before unblocking parallel
  subtests, and after running any cleanup functions.

With this implementation, it should be the case that every test that
is running when a race is detected reports that race, and any race
reported for a subtest is not redundantly reported for its parent.

The regression tests are based on those added in CL 494057 and
CL 501895, with a few additions based on my own review of the code.

Fixes #60083.

Change-Id: I578ae929f192a7a951b31b17ecb560cbbf1ef7a1
Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-linux-amd64-longtest-race,gotip-windows-amd64-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/506300
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
2023-10-25 20:44:25 +00:00
Marcel Meyer 841c3eb166 all: fix typos in go file comments
These typos were found by executing grep, aspell, sort, and uniq in
a pipe and searching the resulting list manually for possible typos.

    grep -r --include '*.go' -E '^// .*$' . | aspell list | sort | uniq

Change-Id: I56281eda3b178968fbf104de1f71316c1feac64f
GitHub-Last-Rev: e91c7cee34
GitHub-Pull-Request: golang/go#57669
Reviewed-on: https://go-review.googlesource.com/c/go/+/460767
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
2023-01-09 15:34:31 +00:00
Russ Cox 30b240b102 testing: implement -cpu and -count for fuzz tests
Fuzz tests are meant to be run just like ordinary tests,
so copy the same loop cpu and count loops used in testing.go
(and benchmark.go) into fuzz.go.

Change-Id: Ic585df8ccc577869c877b1055e0493803dbeb828
Reviewed-on: https://go-review.googlesource.com/c/go/+/443377
Auto-Submit: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
2022-11-02 13:14:08 +00:00
Russ Cox 1c72ee7f13 testing: fix many test2json inaccuracies
Test2json is parsing the output stream from the test, which includes
package testing's own framing lines intermingled with other output,
in particular any output printed via fmt.Printf, println, and so on.
We have had recurring problems with unexpected partial output lines
causing a framing line to be missed.

A recent talk at GopherCon gave an example of an integration test
involving Docker that happened to print \r-terminated lines instead
of \n-terminated lines in some configurations, which in turn broke
test2json badly. (https://www.gophercon.com/agenda/session/944259)

There are also a variety of open reported issues with similar problems,
which this CL also addresses. The general approach is to add a new
testing flag -test.v=json that means to print additional output to help
test2json. And then test2json takes advantage of that output.

Among the fixes:

 - Identify testing framing more reliably, using ^V
   (#23036, #26325, #43683, GopherCon talk)
 - Test that output with \r\n endings is handled correctly
   (#43683, #34286)
 - Use === RUN in fuzz tests (#52636, #48132)
 - Add === RUN lines to note benchmark starts (#27764, #49505)
 - Print subtest --- PASS/FAIL lines as they happen (#29811)
 - Add === NAME lines to emit more test change events,
   such as when a subtest stops and the parent continues running.
 - Fix event shown in overall test failure (#27568)
 - Avoid interleaving of writes to os.Stdout and os.Stderr (#33419)

Fixes #23036.
Fixes #26325.
Fixes #27568.
Fixes #27764.
Fixes #29811.
Fixes #33419.
Fixes #34286.
Fixes #43683.
Fixes #49505.
Fixes #52636.

Change-Id: Id4207b746a20693f92e52d68c6e4a7f8c41cc7c6
Reviewed-on: https://go-review.googlesource.com/c/go/+/443596
Auto-Submit: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-10-26 19:50:36 +00:00
Russ Cox e4a2c38af5 cmd/go, testing: add go test -skip flag
For proposal #41583, add a new 'go test -skip' flag to make it easy
to disable specific tests, benchmarks, examples, or fuzz targets.

Fixes #41583.

Change-Id: Id12a6575f505dafdce4a149aedc454a002e93afa
Reviewed-on: https://go-review.googlesource.com/c/go/+/421439
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
2022-09-16 14:48:54 +00:00
cuiweixie ed530dbd07 testing: use strings.Builder
Change-Id: Ied8a3019f1e21c0e74c798d2b1b279a606a38554
Reviewed-on: https://go-review.googlesource.com/c/go/+/428279
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
2022-09-07 07:03:10 +00:00
cuiweixie a1c9783ca1 testing: convert numFailed to atomic type
Change-Id: Ic3464e95ad8901df5477d7717760b8c6d08ce97b
Reviewed-on: https://go-review.googlesource.com/c/go/+/426078
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
2022-08-29 14:52:44 +00:00
Russ Cox 19309779ac all: gofmt main repo
[This CL is part of a sequence implementing the proposal #51082.
The design doc is at https://go.dev/s/godocfmt-design.]

Run the updated gofmt, which reformats doc comments,
on the main repository. Vendored files are excluded.

For #51082.

Change-Id: I7332f099b60f716295fb34719c98c04eb1a85407
Reviewed-on: https://go-review.googlesource.com/c/go/+/384268
Reviewed-by: Jonathan Amsterdam <jba@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2022-04-11 16:34:30 +00:00
Bryan C. Mills f985833dec testing: panic in Fuzz if the function returns a value
Otherwise, the behavior of a fuzz target that returns an error could
be confusing.

Fuzz is already documented to require a function “with no return
value”, so this fixes the implementation to match the existing
documentation.

Fixes #51222

Change-Id: I44ca7ee10960214c92f5ac066ac8484c8bb9cd6f
Reviewed-on: https://go-review.googlesource.com/c/go/+/386175
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Nooras Saba‎ <saba@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-02-16 16:06:39 +00:00
Roland Shoemaker 1f411e9b6d testing: only snapshot coverage during fuzzing
Only snapshot/reset coverage counters when we are actually fuzzing.
This prevents a race when running corpus/seed values during the testing
phase.

Fixes #50488

Change-Id: I7dd5a0353a296c0b13eede29ad9af7c78814fa2d
Reviewed-on: https://go-review.googlesource.com/c/go/+/376554
Trust: Katie Hockman <katie@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Trust: Roland Shoemaker <roland@golang.org>
Run-TryBot: Roland Shoemaker <roland@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-01-10 21:48:09 +00:00
Katie Hockman b9cae6f78f testing: fix deadlock with t.Parallel in testing seed corpus
The c.startParallel channel on the testContext is stuck
in t.Parallel() because c.running starts at 1 for the main
fuzz parent test, and is causing a deadlock because it is
never released. It would normally be released by tRunner,
but needs to instead be released by fRunner instead for fuzz
tests.

Fixes #50217

Change-Id: I2d010e9adddfd8e8321ff2f9dd2e43daf46c128f
Reviewed-on: https://go-review.googlesource.com/c/go/+/374054
Reviewed-by: Roland Shoemaker <roland@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2022-01-06 22:12:11 +00:00
Dan Kortschak 9901d9e87a all: fix spelling errors found by misspell
Change-Id: Icedd0c3d49259d5aee249ecb33374e9b78e0c275
Reviewed-on: https://go-review.googlesource.com/c/go/+/373376
Reviewed-by: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2021-12-20 23:30:58 +00:00
ichxxx 567b177949 all: fix typo in comment
Remove duplicate 'the'

Change-Id: I3ed81c8d9c488662387e45580a3bcd462448ba44
GitHub-Last-Rev: 86443993b9
GitHub-Pull-Request: golang/go#50017
Reviewed-on: https://go-review.googlesource.com/c/go/+/372394
Reviewed-by: Robert Griesemer <gri@golang.org>
Run-TryBot: Robert Griesemer <gri@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
2021-12-15 19:56:22 +00:00
Katie Hockman f909f813a0 testing: update docs for fuzzcachedir
Although most of the code seems to be already implemented
to support this for general use, it didn't make it in for
Go 1.18, so for now we should at least document that it's
only for use by the go command.

Change-Id: Id559e72d590aedeaaa50bcf880bca1a385d858dd
Reviewed-on: https://go-review.googlesource.com/c/go/+/370954
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
2021-12-13 18:55:52 +00:00
Russ Cox 2580d0e08d all: gofmt -w -r 'interface{} -> any' src
And then revert the bootstrap cmd directories and certain testdata.
And adjust tests as needed.

Not reverting the changes in std that are bootstrapped,
because some of those changes would appear in API docs,
and we want to use any consistently.
Instead, rewrite 'any' to 'interface{}' in cmd/dist for those directories
when preparing the bootstrap copy.

A few files changed as a result of running gofmt -w
not because of interface{} -> any but because they
hadn't been updated for the new //go:build lines.

Fixes #49884.

Change-Id: Ie8045cba995f65bd79c694ec77a1b3d1fe01bb09
Reviewed-on: https://go-review.googlesource.com/c/go/+/368254
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
2021-12-13 18:45:54 +00:00
sivchari 7e1260f62b testing: simplify fuzzResult.String to avoid unnecessarily using fmt.Sprintf
Change-Id: I16b6bfb6b0f02672c894b20845aa14d8dd1979b4
GitHub-Last-Rev: 75ab90123a
GitHub-Pull-Request: golang/go#49819
Reviewed-on: https://go-review.googlesource.com/c/go/+/367314
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com>
TryBot-Result: Go Bot <gobot@golang.org>
2021-11-27 23:25:39 +00:00
Katie Hockman 9d89a5eb64 all: update terminology for fuzzing
This change doesn't modify any functionality.
It also doesn't update all of the comments and
variable names of the internal code, but everything
user facing should be correct.

Updates #49185

Change-Id: Ia8b2c94b89ba45897c4085ea0c17a3d8896f7ec7
Reviewed-on: https://go-review.googlesource.com/c/go/+/362794
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
2021-11-11 21:06:07 +00:00
Katie Hockman 2559a98a3c testing: remove package from fuzz crasher message
Fixes #48149

Change-Id: Iaf91d2c54fda809c7da90cdfb6d1d075f474c69b
Reviewed-on: https://go-review.googlesource.com/c/go/+/362116
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Julie Qiu <julie@golang.org>
2021-11-09 15:16:57 +00:00
Katie Hockman 982060203c testing: don't allow f.Log/Logf or f.Skipped inside f.Fuzz
This change also does some refactors around how
we prevent many (*F) methods from being called
inside (*F).Fuzz. Previously, there was a lot of
comment/code duplication, which was going to be
difficult to maintain and brittle. The refactor
lessens this duplication.

Previously, the methods Log, Logf, Failed, Name and
Skipped were the only (*common) methods that were
allowed to be called inside (*F).Fuzz. After this
change, Failed and Name are still allowed, but
Log, Logf, and Skipped are not (t.Log, t.Logf, or
t.Skipped should be used instead).

Fixes #48988

Change-Id: I4066247d551ea1908e8a2ca2889509fc68e3bb44
Reviewed-on: https://go-review.googlesource.com/c/go/+/356151
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-10-19 18:18:37 +00:00
Katie Hockman 067d796549 testing: write output to buffer when fuzzing
Fixes #48709

Change-Id: Ia6376a2f792946498d6565a53605b3e6c985ea7c
Reviewed-on: https://go-review.googlesource.com/c/go/+/355909
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2021-10-19 15:48:56 +00:00
Jay Conrod d032b2b2c8 testing: don't create unique subtest names while fuzzing
T.Run uses a map[string]int64 to keep track of subtest names that may
be returned through T.Name. T.Name can't return duplicate names for
subtests started with T.Run.

If a fuzz target calls T.Run, this map takes a large amount of memory,
since there are a very large number of subtests that would
otherwise have duplicate names, and the map stores one entry per subtest.
The unique suffixes are not useful (and may be confusing) since the
full sequence of tests cannot be re-run deterministically.

This change deletes all entries in the map before each call to the
function being fuzzed. There is a slight change in the contract of
T.Name while fuzzing.

This change was discussed in CL 351452.

Fixes #44517

Change-Id: I3093a2e5568099ce54aca1006fac84a6fd2c3ddf
Reviewed-on: https://go-review.googlesource.com/c/go/+/354430
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
2021-10-12 20:20:41 +00:00
Katie Hockman 36a265a625 testing: fix -run behavior with fuzz tests
This change fixes some issues with -run, and
the subsequent command line output when running
in verbose mode. It replaces CorpusEntry.Name
with CorpusEntry.Path, and refactors the code
accordingly.

This change also adds a lot of additional tests
which check explicit command line output when
fuzz targets are run without fuzzing. This will
be important to avoid regressions.

Updates #48149

Change-Id: If34b1f51db646317b7b51c3c38ae53231d01f568
Reviewed-on: https://go-review.googlesource.com/c/go/+/354632
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-10-12 14:32:53 +00:00
Katie Hockman 0b4d4998d5 testing: document f.Fuzz requirement to not change underlying data
Updates #48606

Change-Id: I6d555fdefccd842fb65ec8d630b4808bcb54a825
Reviewed-on: https://go-review.googlesource.com/c/go/+/353977
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-10-05 20:15:01 +00:00
Jay Conrod d60ad1e068 testing: address feedback for dev.fuzz merge
Based on comments in CL 348469.

Note that with this change, F.Fuzz no longer calls
runtime.Goexit. This simplifies our logic and makes F.Fuzz more
predictable.

Change-Id: I6c3c65b0e8e8f261621cbe2f17375e8164ef60a0
Reviewed-on: https://go-review.googlesource.com/c/go/+/351316
Trust: Jay Conrod <jayconrod@google.com>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
2021-09-24 21:42:31 +00:00
Roland Shoemaker ccf140f3d7 internal/fuzz: allocate memory for mutated strings
Rather than directly pointing at the underlying scratch slice, allocate
memory for strings. This prevents mutation of previous values we've
passed to the fuzz function, which may be retained by something that
expects them to be immutable.

Fixes golang/go#48308

Change-Id: Iee9bed1a536fdc4188180e8e7c1c722f641271d2
Reviewed-on: https://go-review.googlesource.com/c/go/+/351312
Trust: Roland Shoemaker <roland@golang.org>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Roland Shoemaker <roland@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Katie Hockman <katie@golang.org>
2021-09-22 15:38:34 +00:00
Katie Hockman 24e25afff4 [dev.fuzz] Revert "[dev.fuzz] testing: convert seed corpus values where possible"
This reverts commit 413c125da3.

Reason for revert: Giving this more thought, we've decided that
converting types under the hood may cause unexpected behavior to
users. This is a feature that can always be added after more
consideration has been done, but is not something that can be
removed due to the backwards compatibility promise.

Updates golang/go#45593

Change-Id: I79bab24979d7e4c294e6cb6455d4c7729d6a0efb
Reviewed-on: https://go-review.googlesource.com/c/go/+/350251
Trust: Katie Hockman <katie@golang.org>
Trust: Joe Tsai <joetsai@digital-static.net>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Joe Tsai <joetsai@digital-static.net>
2021-09-20 17:18:57 +00:00
Katie Hockman 5ed7dd0650 [dev.fuzz] internal/fuzz: rework default test behavior before fuzzing
This change refactors some of the code to support skipping a run
of the seed corpus by the go command before runFuzzing occurs.
Previously, the go command would run all seed corpus for all targets
that match the provided `run` argument. This will be redundant when
fuzzing a target. Now, the seed corpus is only run by targets other than
the one that's about to be fuzzed, and the worker handles running and
reporting issues with the seed corpus.

Part of the logic that needed close inspection is what to do if a
failure occurs during a testing-only or coverage-only fail. If the input
is already in the seed corpus, the fuzzing engine shouldn't add it. If
the input is currently in the cache, then it should be written to
testdata. In all cases, if an error occurs, we need to report this to
the user with enough information for them to debug it.

This uncovered some issues with our code when fuzzing without
instrumentation, and when -run=None was provided. There are some logic
fixes in this change, and some small refactors.

Fixes golang/go#48327
Fixes golang/go#48296

Change-Id: I9ce2be0219c5b09277ddd308df8bc5a46d4558fa
Reviewed-on: https://go-review.googlesource.com/c/go/+/349630
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-09-16 13:37:34 +00:00
Katie Hockman 739a42c199 [dev.fuzz] testing: adjust -fuzz multiple match stdout
Fixes golang/go#48131

Change-Id: I40ff130c849dffe38363ddc0282e93ceb74ae140
Reviewed-on: https://go-review.googlesource.com/c/go/+/349969
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-09-15 18:52:34 +00:00
Jay Conrod c8dfa306ba [dev.fuzz] testing: F.Setenv plus various fixes and revisions
I spent some time looking through all the changes we've made to
testing and cmd/go/... on the dev.fuzz branch. CL 348469 shows those
differences. This CL fixes comments, TODOs, and simplifies code in a
few places. It also implements F.Setenv.

Change-Id: I6fd7ef5fbd0bb6055e38d56cb42bddcf6f4ffdaf
Reviewed-on: https://go-review.googlesource.com/c/go/+/349109
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Jay Conrod <jayconrod@google.com>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
2021-09-10 21:28:48 +00:00
Katie Hockman 363f2f3df9 [dev.fuzz] testing: allow -fuzzminimizetime to be 0
Fixes golang/go#48321

Change-Id: I1547379eb7a703f7f3c4594d27833eb3587796a0
Reviewed-on: https://go-review.googlesource.com/c/go/+/349089
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-09-10 17:46:52 +00:00
Katie Hockman b38e853d04 [dev.fuzz] testing: move corpus from testdata/corpus to testdata/fuzz
Fixes golang/go#48128

Change-Id: I04deb17226ff5d5dfcbe408fe089f400d4d3836f
Reviewed-on: https://go-review.googlesource.com/c/go/+/349053
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-09-10 15:20:44 +00:00
Roland Shoemaker e5247f7886 [dev.fuzz] internal/fuzz: don't store corpus in memory
Instead of holding all corpus data/values in memory, only store seed
inputs added via F.Add in memory, and only load corpus entries which
are written to disk when we need them. This should significantly reduce
the memory required by the coordinator process.

Additionally only load the corpus in the coordinator process, since the
worker has no need for it.

Fixes #46669.

Change-Id: Ic3b0c5e929fdb3e2877b963e6b0fa14e140c1e1d
Reviewed-on: https://go-review.googlesource.com/c/go/+/345096
Trust: Roland Shoemaker <roland@golang.org>
Run-TryBot: Roland Shoemaker <roland@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-09-02 21:43:52 +00:00
Jay Conrod f0668e7c8c [dev.fuzz] cmd/go: stream output when fuzzing
Previously, 'go test' streamed output when there were no package
arguments or when benchmarking. This CL expands that to include
fuzzing to ensure that coordinator progress messages are printed.

This change tweaks tests and output a little bit: the output is
slightly different depending on whether it was streamed or buffered in
'go test'.

Fixes golang/go#47603

Change-Id: I387470062cf0620f5c7f214b6f54039c921912c4
Reviewed-on: https://go-review.googlesource.com/c/go/+/344831
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Jay Conrod <jayconrod@google.com>
Reviewed-by: Katie Hockman <katie@golang.org>
2021-09-01 22:11:17 +00:00
Katie Hockman 0cf1e16bac [dev.fuzz] all: merge master (c95464f) into dev.fuzz
The new SetEnv method for *testing.T and *testing.B types
was automatically supported by *testing.F since it was added
to the *testing.common type. This function is not appropriate
for *testing.F since fuzzing is run in parallel by default.

Conflicts:

- api/next.txt

Merge List:

+ 2021-06-27 c95464f0ea internal/buildcfg: refactor GOEXPERIMENT parsing code somewhat
+ 2021-06-25 ed01ceaf48 runtime/race: use race build tag on syso_test.go
+ 2021-06-25 d1916e5e84 go/types: in TestCheck/issues.src, import regexp/syntax instead of cmd/compile/internal/syntax
+ 2021-06-25 5160896c69 go/types: in TestStdlib, import from source instead of export data
+ 2021-06-25 d01bc571f7 runtime: make ncgocall a global counter
+ 2021-06-25 37f9a8f69d go/types: fix a bug in package qualification logic
+ 2021-06-24 c309c89db5 reflect: document that InterfaceData is a low-entropy RNG
+ 2021-06-24 cce621431a cmd/compile: fix wrong type in SSA generation for OSLICE2ARRPTR
+ 2021-06-24 600a2a4ffb cmd/go: don't try to add replaced versions that won't be selected
+ 2021-06-24 a9bb38222a net: remove hard-coded timeout in dialClosedPort test helper
+ 2021-06-24 86d72fa2cb time: handle invalid UTF-8 byte sequences in quote to prevent panic
+ 2021-06-24 44a12e5f33 cmd/go: search breadth-first instead of depth-first for test dependency cycles
+ 2021-06-24 73496e0df0 net: use absDomainName in the Windows lookupPTR test helper
+ 2021-06-24 222ed1b38a os: enable TestFifoEOF on openbsd
+ 2021-06-22 0ebd5a8de0 cmd/go: update ToolTags based on GOARCH value
+ 2021-06-22 5bd09e5efc spec: unsafe.Add/Slice are not permitted in statement context
+ 2021-06-22 666315b4d3 runtime/internal/atomic: remove incorrect pointer indirection in comment
+ 2021-06-22 63daa774b5 go/types: guard against checking instantiation when generics is disabled
+ 2021-06-22 197a5ee2ab cmd/gofmt: remove stale documentation for the -G flag
+ 2021-06-22 9afd158eb2 go/parser: parse an ast.IndexExpr for a[]
+ 2021-06-21 1bd5a20e3c cmd/go: add a -go flag to 'go mod graph'
+ 2021-06-21 761edf71f6 cmd/internal/moddeps: use a temporary directory for GOMODCACHE if needed
+ 2021-06-21 a0400420ad cmd/internal/moddeps: use -mod=readonly instead of -mod=mod
+ 2021-06-21 3f9ec83b10 cmd/go: document GOPPC64 environment variable
+ 2021-06-21 20bdfba325 go/scanner: fall back to next() when encountering 0 bytes in parseIdentifier
+ 2021-06-21 44f9a3566c database/sql: fix deadlock test in prepare statement
+ 2021-06-21 16e82be454 runtime: fix crash during VDSO calls on PowerPC
+ 2021-06-21 2e542c3c06 runtime/pprof: deflake TestMorestack more
+ 2021-06-21 ced0fdbad0 doc/go1.17: note deprecation of 'go get' for installing commands
+ 2021-06-21 7a5e7047a4 doc/go1.17: add Go 1.18 pre-announcements
+ 2021-06-21 85a2e24afd doc/go1.17: add security-related release notes
+ 2021-06-21 1de332996c doc/go1.17: document go/parser.SkipObjectResolution
+ 2021-06-21 117ebe0f52 cmd/go: do not require the module cache to exist for 'go mod edit'
+ 2021-06-20 460900a7b5 os/signal: test with a significantly longer fatal timeout
+ 2021-06-19 b73cc4b02b database/sql: do not rely on timeout for deadlock test
+ 2021-06-18 86743e7d86 image: add RGBA64Image interface
+ 2021-06-18 9401172166 runtime: clarify Frames.Next documentation
+ 2021-06-18 57aaa19aae runtime: disable CPU profiling before removing the SIGPROF handler
+ 2021-06-18 6f22d2c682 doc/go1.17: fix typo
+ 2021-06-17 45f251ad6c cmd/pprof,runtime/pprof: disable test on more broken platforms
+ 2021-06-17 ed834853ad cmd/go: replace a TODO with an explanatory comment
+ 2021-06-17 4dede02550 cmd/pprof: make ObjAddr a no-op
+ 2021-06-17 97cee43c93 testing: drop unusual characters from TempDir directory name
+ 2021-06-17 b0355a3e72 time: fix receiver for Time.IsDST method
+ 2021-06-17 881b6ea7ba doc/go1.17: fix redundant space
+ 2021-06-16 0e67ce3d28 cmd/go: in lazy modules, add transitive imports for 'go get' arguments
+ 2021-06-16 6ea2af0890 cmd/go: add a regression test for #45979
+ 2021-06-16 a294e4e798 math/rand: mention half-open intervals explicitly
+ 2021-06-16 a6a853f94c cmd/asm: restore supporting of *1 scaling on ARM64
+ 2021-06-16 785a8f677f cmd/compile: better error message for invalid untyped operation
+ 2021-06-16 a752bc0746 syscall: fix TestGroupCleanupUserNamespace test failure on Fedora
+ 2021-06-15 d77f4c0c5c net/http: improve some server docs
+ 2021-06-15 219fe9d547 cmd/go: ignore UTF8 BOM when reading source code
+ 2021-06-15 723f199edd cmd/link: set correct flags in .dynamic for PIE buildmode
+ 2021-06-15 4d2d89ff42 cmd/go, go/build: update docs to use //go:build syntax
+ 2021-06-15 033d885315 doc/go1.17: document go run pkg@version
+ 2021-06-15 ea8612ef42 syscall: disable c-shared test when no cgo, for windows/arm
+ 2021-06-15 abc56fd1a0 internal/bytealg: remove duplicate go:build line
+ 2021-06-15 4061d3463b syscall: rewrite handle inheritance test to use C rather than Powershell
+ 2021-06-15 cf4e3e3d3b reflect: explain why convertible or comparable types may still panic
+ 2021-06-14 7841cb14d9 doc/go1.17: assorted fixes
+ 2021-06-14 8a5a6f46dc debug/elf: don't apply DWARF relocations for ET_EXEC binaries
+ 2021-06-14 9d13f8d43e runtime: update the variable name in comment
+ 2021-06-14 0fd20ed5b6 reflect: use same conversion panic in reflect and runtime
+ 2021-06-14 6bbb0a9d4a cmd/internal/sys: mark windows/arm64 as c-shared-capable
+ 2021-06-14 d4f34f8c63 doc/go1.17: reword "results" in stack trace printing
+ 2021-06-14 fdab5be159 doc/go1.17: further revise OpenBSD release notes
+ 2021-06-14 326ea438bb cmd/compile: rewrite a, b = f() to use temporaries when type not identical
+ 2021-06-14 3249b645c9 cmd/compile: factor out rewrite multi-valued f()
+ 2021-06-13 14305bf0b9 misc/cgo: generate Windows import libraries for clang
+ 2021-06-13 24cff0f044 cmd/go, misc/cgo: skip test if no .edata
+ 2021-06-13 67b1b6a2e3 cmd/compile: allow ir.OSLICE2ARRPTR in mayCall
+ 2021-06-12 1ed0d129e9 runtime: testprogcgo: don't call exported Go functions directly from Go
+ 2021-06-12 9d46ee5ac4 reflect: handle stack-to-register translation in callMethod
+ 2021-06-11 e552a6d312 cmd/go: remove hint when no module is suggested
+ 2021-06-11 16b5d766d8 syscall: do not load native libraries on non-native powershell on arm
+ 2021-06-11 77aa209b38 runtime: loop on EINTR in macOS sigNoteSleep
+ 2021-06-11 e2dc6dd5c9 doc/go1.17: clean up formatting of gofmt section
+ 2021-06-11 2f1128461d cmd/go: match Windows paths in TestScript/mod_invalid_version
+ 2021-06-11 2721da2608 doc/go1.17: fix formatting near httptest
+ 2021-06-10 770f1de8c5 net/http: remove test-only private key from production binaries
+ 2021-06-10 8d11b1d117 cmd/go: report the imports of CompiledGoFiles in ImportMap
+ 2021-06-10 dc00dc6c6b crypto/tls: let HTTP/1.1 clients connect to servers with NextProtos "h2"
+ 2021-06-09 27f83723e9 api: promote next to go1.17
+ 2021-06-09 182157c81a doc/go1.17: remove lingering TODO
+ 2021-06-09 a5bc060b42 doc/go1.17: document strconv changes for Go 1.17
+ 2021-06-09 1402b27d46 strconv: document parsing of leading +/-
+ 2021-06-09 df35ade067 doc/go1.17: document //go:build lines
+ 2021-06-09 e4e7807d24 net/http: add AllowQuerySemicolons
+ 2021-06-09 ec3026d032 doc/go1.17: remove TODO for ports section
+ 2021-06-09 e6dda19888 net/url: reject query values with semicolons
+ 2021-06-09 139e935d3c math/big: comment division
+ 2021-06-09 aa5540cd82 cmd/compile: make map.zero symbol content-addressable
+ 2021-06-09 07ca28d529 cmd/link: fix bug in -strictdups checking of BSS symbols
+ 2021-06-08 bcecae2af6 doc/go1.17: mention new possibility of type conversion panicking
+ 2021-06-08 63dcab2e91 doc/go1.17: mention new vet checks sigchanyzer and stdmethods.
+ 2021-06-08 6551763a60 doc/go1.17: mention block profile bias fix
+ 2021-06-08 cb80937bf6 Revert "doc/go1.17: mention block profile bias fix"
+ 2021-06-08 d3e3d03666 net: reject leading zeros in IP address parsers
+ 2021-06-08 da4a640141 doc/go1.17: revise OpenBSD release notes
+ 2021-06-08 689f4c7415 doc/go1.17: mention block profile bias fix
+ 2021-06-08 9afe071c60 doc/go1.17: remove TODO for Tools section
+ 2021-06-08 f753d7223e doc/go1.17: resolve TODO for cmd/cover
+ 2021-06-08 9498b0155d cmd/go: in Go 1.17+ modules, add indirect go.mod dependencies separately from direct ones
+ 2021-06-08 949f00cebe doc/go1.17: add release notes for crypto packages
+ 2021-06-08 0fb3e2c184 doc/go1.17: add a release note for the '-compat' flag to 'go mod tidy'
+ 2021-06-08 2169deb352 cmd/compile: use t.AllMethods when sorting typesByString
+ 2021-06-08 c20bcb6488 runtime: remove out-of-date comments about frame skipping
+ 2021-06-07 39c39ae52f doc: document Go 1.17 language changes
+ 2021-06-07 dc8b558951 cmd/dist: pass -Wno-lto-type-mismatch in swig_callback_lto
+ 2021-06-07 909dd5e010 strconv: ParseFloat: always return ErrSyntax for bad syntax
+ 2021-06-07 8212707871 crypto/elliptic: update P-521 docs to say it's constant-time
+ 2021-06-07 7406180012 fmt: split package documentation into more sections
+ 2021-06-07 e3176bbc3e crypto/tls: fix typo in Config.NextProtos docs
+ 2021-06-05 e1fa26026d spec: improve wording consistency by eliminating "specifier"
+ 2021-06-05 f490134126 spec: improve wording by choosing an official term "keyword"
+ 2021-06-05 e3cb381704 go/internal/gcimporter: don't waste CPU copying bytes in `io.ReadAll`
+ 2021-06-05 9d669ed47a misc/cgo/errors: use expected column numbers
+ 2021-06-04 95939e8de7 cmd/compile/internal/abi: fix typo in comment
+ 2021-06-04 831f9376d8 net/http: fix ResponseWriter.ReadFrom with short reads
+ 2021-06-04 3a9d906edc os: avoid finalizer race in windows process object
+ 2021-06-04 105c5b50e0 os: terminate windows processes via handle directly
+ 2021-06-04 79cd407f88 syscall: regenerate zsyscall_windows.go
+ 2021-06-04 c6b6211229 doc/go1.17: document testing changes for Go 1.17
+ 2021-06-04 0214440075 syscall: do not pass console handles to PROC_THREAD_ATTRIBUTE_HANDLE_LIST on Windows 7
+ 2021-06-04 962d5c997a cmd/compile,go/types: restrict use of unsafe.{Add,Slice} to go1.17 or newer
+ 2021-06-04 b29b123e07 cmd/compile: remove spurious ir.Dump
+ 2021-06-03 6d98301114 cmd/link: use correct alignment in PE DWARF sections
+ 2021-06-03 e0d029f758 runtime: avoid gp.lockedm race in exitsyscall0

Change-Id: I00216c3c36e64814c44c79f25d1f38e4df6c1f24
2021-06-28 13:25:00 -04:00
Katie Hockman 1cfa89e5dc [dev.fuzz] internal/fuzz: use scratch []byte for mutations
The mutator will now use a scratch []byte when mutating
[]byte and string types.

I ran the following target locally:

    func FuzzBytesFromStringCmp(f *testing.F) {
        f.Fuzz(func(t *testing.T, a, b string) {
            bytes.Compare([]byte(a), []byte(b))
        })
    }

Before the change, execs were <400/sec:

    === FUZZ  FuzzBytesFromStringCmp
    fuzzing, elapsed: 3.0s, execs: 1090 (363/sec), workers: 8 ...
    fuzzing, elapsed: 6.0s, execs: 2290 (382/sec), workers: 8 ...
    fuzzing, elapsed: 9.0s, execs: 3491 (388/sec), workers: 8 ...
    fuzzing, elapsed: 12.0s, execs: 4691 (391/sec), workers: 8 ...
    fuzzing, elapsed: 15.0s, execs: 5869 (391/sec), workers: 8 ...
    fuzzing, elapsed: 18.0s, execs: 7056 (392/sec), workers: 8 ...

After the change, the execs are ~6000/sec

    === FUZZ  FuzzBytesFromStringCmp
    fuzzing, elapsed: 3.0s, execs: 155129 (51687/sec), workers: 8 ...
    fuzzing, elapsed: 6.0s, execs: 303710 (50606/sec), workers: 8 ...
    fuzzing, elapsed: 9.0s, execs: 454314 (50470/sec), workers: 8 ...
    fuzzing, elapsed: 12.0s, execs: 603212 (50262/sec), workers: 8 ...
    fuzzing, elapsed: 15.0s, execs: 756165 (50401/sec), workers: 8 ...
    fuzzing, elapsed: 18.0s, execs: 899293 (49955/sec), workers: 8 ...

Which is comparable to the same target with two []byte as input:

    === FUZZ  FuzzBytesCmp
    fuzzing, elapsed: 3.0s, execs: 152348 (50757/sec), workers: 8 ...
    fuzzing, elapsed: 6.0s, execs: 314386 (52387/sec), workers: 8 ...
    fuzzing, elapsed: 9.0s, execs: 487413 (54148/sec), workers: 8 ...
    fuzzing, elapsed: 12.0s, execs: 646886 (53901/sec), workers: 8 ...
    fuzzing, elapsed: 15.0s, execs: 814257 (54266/sec), workers: 8 ...
    fuzzing, elapsed: 18.0s, execs: 983214 (54619/sec), workers: 8 ...

Benchark results:
name                      old time/op    new time/op    delta
MutatorBytes/1-8          7.70ms ± 3%    0.00ms ± 3%   -99.99%  (p=0.029 n=4+4)
MutatorBytes/10-8         7.88ms ± 2%    0.00ms ± 6%   -99.99%  (p=0.029 n=4+4)
MutatorBytes/100-8        7.87ms ± 1%    0.00ms ± 2%   -99.99%  (p=0.029 n=4+4)
MutatorBytes/1000-8       8.11ms ± 5%    0.00ms ± 2%   -99.99%  (p=0.029 n=4+4)
MutatorBytes/10000-8      8.11ms ± 4%    0.00ms ± 2%   -99.99%  (p=0.029 n=4+4)
MutatorBytes/100000-8     8.28ms ±10%    0.00ms ± 4%   -99.96%  (p=0.029 n=4+4)
MutatorString/1-8         7.89ms ± 5%    0.00ms ±17%   -99.99%  (p=0.029 n=4+4)
MutatorString/10-8        7.91ms ± 4%    0.00ms ± 7%   -99.99%  (p=0.029 n=4+4)
MutatorString/100-8       8.08ms ± 4%    0.00ms ± 7%   -99.99%  (p=0.029 n=4+4)
MutatorString/1000-8      8.11ms ± 6%    0.00ms ±11%   -99.99%  (p=0.029 n=4+4)
MutatorString/10000-8     8.04ms ± 7%    0.00ms ± 8%   -99.98%  (p=0.029 n=4+4)
MutatorString/100000-8    8.24ms ± 7%    0.01ms ±13%   -99.82%  (p=0.029 n=4+4)

Fixes #46543

Change-Id: I8b078ed3adc1bb6310c33afc49bb6cd78e7e976c
Reviewed-on: https://go-review.googlesource.com/c/go/+/324849
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-06-28 16:34:17 +00:00
Jay Conrod cc04ab463d [dev.fuzz] testing: report T.Deadline when running seed values
T.Deadline should return the test deadline, set with -timeout. When
fuzz targets are run with seed values as unit tests, either with or
without -fuzz, T.Deadline should work inside the fuzz function.

There is no deadline when fuzzing, even if -fuzztime is set, since
workers may have much shorter deadlines, and fuzz function behavior
shouldn't be time-dependent anyway.

Fixes #46220

Change-Id: I84aaeb9d7bfdc12bdcb6f1ab3fe67b3067ad2dfe
Reviewed-on: https://go-review.googlesource.com/c/go/+/330509
Trust: Jay Conrod <jayconrod@google.com>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
2021-06-24 17:45:48 +00:00
Roland Shoemaker df99a270b3 [dev.fuzz] testing: capture coverage even if tRunner failed
Call SnapshotCoverage when the fuzzFn panics, or t.Fatal(f) is called.
We currently don't use this coverage for anything, but it is necessary
to allow the coordinator to continue when loading a corpus that contains
an input that causes crashes. We will also probably want this behavior
once we allow the fuzzer to continue after finding a crasher, since the
input used to find one crasher may be a useful input for further
mutation.

Fixes #46633

Change-Id: I40ed5440c88fa354d90a4ff4ae2bf8a19bf3254f
Reviewed-on: https://go-review.googlesource.com/c/go/+/328650
Trust: Roland Shoemaker <roland@golang.org>
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Roland Shoemaker <roland@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Katie Hockman <katie@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
2021-06-23 22:19:47 +00:00
Katie Hockman 413c125da3 [dev.fuzz] testing: convert seed corpus values where possible
The types provided in f.Fuzz will be viewed as the
canonical types for fuzzing. If the type is different
for a seed corpus entry, then the testing package
will attempt to convert it. If it can't convert it,
f.Fuzz will fail.

Currently, this allows converting types that may result
in precision loss or a semantically different value.
For example, an int(-1) can be converted to uint even
though the value could be math.MaxUint64. There is a
TODO to consider improving this in the future.

Updates golang/go#45593

Change-Id: I2e752119662f46b68445d42b1ffa46dd30e9faea
Reviewed-on: https://go-review.googlesource.com/c/go/+/325702
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
2021-06-15 19:32:21 +00:00
Katie Hockman b2ff3e82ba [dev.fuzz] testing: fix documentation for fuzzminimizetime
This removes redundancy for the default value.

Fixes #46555

Change-Id: Ib62bd2d584ef82bef806d0fe2ce59957488e469e
Reviewed-on: https://go-review.googlesource.com/c/go/+/325070
Trust: Katie Hockman <katie@golang.org>
Run-TryBot: Katie Hockman <katie@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Roland Shoemaker <roland@golang.org>
2021-06-07 20:47:57 +00:00