This CL adds a new flag to the testing package and the go test command
which randomizes the execution order for tests and benchmarks.
This can be useful for identifying unwanted dependencies
between test or benchmark functions.
The flag is off by default. If `-shuffle` is set to `on` then the system
clock will be used as the seed value. If `-shuffle` is set to an integer
N, then N will be used as the seed value. In both cases, the seed will
be reported for failed runs so that they can reproduced later on.
Fixes#28592
Change-Id: I62e7dfae5f63f97a0cbd7830ea844d9f7beac335
Reviewed-on: https://go-review.googlesource.com/c/go/+/310033
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Trust: Bryan C. Mills <bcmills@google.com>
Also add a test to testing/fstest.
Fixes#45186
Change-Id: I00e5f46ccd5269dbc266a8f2ebc9a62ebb1297b8
Reviewed-on: https://go-review.googlesource.com/c/go/+/311649
Trust: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
Protects the usages of (*common).finished with locks
to prevent data races, thus allowing benchmarks to safely invoke
.Fatal* and .Skip* concurrently.
Fixes#45526
Change-Id: I2b4846f525c426d6c7d3418f8f6c86446adbf986
Reviewed-on: https://go-review.googlesource.com/c/go/+/309572
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Update references missed in CL 263142.
For #41190
Change-Id: I778760a6a69bd0440fec0848bdef539c9ccb4ee1
GitHub-Last-Rev: dda42b09ff
GitHub-Pull-Request: golang/go#42874
Reviewed-on: https://go-review.googlesource.com/c/go/+/273946
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Cherry Zhang <cherryyz@google.com>
Change-Id: I2ed394fccec83aac09a49673554cbf504787965b
Reviewed-on: https://go-review.googlesource.com/c/go/+/306810
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Trust: Dmitri Shuralyov <dmitshur@golang.org>
parent's helperNames has not been set when frameSkip called, moving
helperNames initilazing to frameSkip.
Fixes#44887
Change-Id: I5107c5951033e5e47d1ac441eac3ba5344a7bdc0
GitHub-Last-Rev: 44b90b2e2e
GitHub-Pull-Request: golang/go#45071
Reviewed-on: https://go-review.googlesource.com/c/go/+/302469
Trust: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
The sentence starts "fsys must only contain",
which leads the reader to believe that fsys must not contain others.
The rapid reversal leads to confusion.
I had to read it several times to be sure I'd parsed it correctly.
Remove "only"; rely on the rest of the sentence to clarify.
Change-Id: I9fb7935aed4f9839344d3a00b761d20981fba864
Reviewed-on: https://go-review.googlesource.com/c/go/+/296529
Trust: Josh Bleecher Snyder <josharian@gmail.com>
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Add a new method TB.Setenv that'll set environment variables
only for the isolated lifetime of the test, and will clean up
and unset these variables when the test ends.
This method disables the test or benchmark from running in
parallel.
Fixes#41260
Change-Id: I0a18f094ec1c6ec3157b4b12993ea3075e2e9867
GitHub-Last-Rev: 0ca12fa565
GitHub-Pull-Request: golang/go#41857
Reviewed-on: https://go-review.googlesource.com/c/go/+/260577
Trust: Daniel Martí <mvdan@mvdan.cc>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: roger peppe <rogpeppe@gmail.com>
"[-]" is not a valid path.Match pattern.
Fixes#44474
Change-Id: I0932bbf08ffb8ad0c5337d69d0893f53c1ba89ad
Reviewed-on: https://go-review.googlesource.com/c/go/+/294869
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
When you log after a test has completed,
the testing package panics.
Print the logged line as part of that panic,
to aid in debugging.
Change-Id: I3d6689d1eed57c03e300afe37db0c15b2f4acda4
Reviewed-on: https://go-review.googlesource.com/c/go/+/283972
Trust: Josh Bleecher Snyder <josharian@gmail.com>
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Make all our package sources use Go 1.17 gofmt format
(adding //go:build lines).
Part of //go:build change (#41184).
See https://golang.org/design/draft-gobuild
Change-Id: Ia0534360e4957e58cd9a18429c39d0e32a6addb4
Reviewed-on: https://go-review.googlesource.com/c/go/+/294430
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Jason A. Donenfeld <Jason@zx2c4.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Do not require directory entry and Stat result to match for symlinks,
because they won't (Stat dereferences the symlink).
Fixes#44113.
Change-Id: Ifc6dbce5719906e2f42254a7172f1ef787464a9e
Reviewed-on: https://go-review.googlesource.com/c/go/+/290009
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
The testing.TestFS function assumes that the file system it's testing
doesn't change under it. Clarify this in the documentation and fix the
use of os.TestDirFS that's currently susceptible to this race.
Fixes#42637.
Change-Id: Ia7792380726177f8953d150ee87381b66cb01cb3
Reviewed-on: https://go-review.googlesource.com/c/go/+/282452
Trust: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
os.ReadDir is a replacement for ioutil.ReadDir that returns
a slice of fs.DirEntry instead of fs.FileInfo, meaning it is the
more efficient form.
This CL updates call sites throughout the Go source tree
wherever possible. As usual, code built using the Go 1.4
bootstrap toolchain is not included. There is also a use in
go/build that appears in the public API and can't be changed,
at least not without additional changes.
Fixes#42026.
Change-Id: Icfc9dd52c6045020f6830e22c72128499462d561
Reviewed-on: https://go-review.googlesource.com/c/go/+/266366
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
As part of #42026, these helpers from io/ioutil were moved to os.
(ioutil.TempFile and TempDir became os.CreateTemp and MkdirTemp.)
Update the Go tree to use the preferred names.
As usual, code compiled with the Go 1.4 bootstrap toolchain
and code vendored from other sources is excluded.
ReadDir changes are in a separate CL, because they are not a
simple search and replace.
For #42026.
Change-Id: If318df0216d57e95ea0c4093b89f65e5b0ababb3
Reviewed-on: https://go-review.googlesource.com/c/go/+/266365
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Sub provides a convenient way to refer to a subdirectory
automatically in future operations, like Unix's chdir(2).
The CL also includes updates to fstest to check Sub implementations.
As part of updating fstest, I changed the meaning of TestFS's
expected list to introduce a special case: if you list no expected files,
that means the FS must be empty. In general it's OK not to list all
the expected files, but if you list none, that's almost certainly a
mistake - if your FS were broken and empty, you wouldn't find out.
Making no expected files mean "must be empty" makes the mistake
less likely - if your file system ever worked, then your test will keep
it working.
That change found a testing bug: embedtest was making exactly
that mistake.
Fixes#42322.
Change-Id: I63fd4aa866b30061a0e51ca9a1927e576d6ec41e
Reviewed-on: https://go-review.googlesource.com/c/go/+/274856
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Store the PC instead of the string name of the function, and defer
that conversion until we need it.
Helper is still relatively expensive in CPU time (few hundred ns),
but memory allocation is now constant for a test rather than linear in
the number of times Helper is called.
benchstat:
name old time/op new time/op delta
TBHelper-4 1.30µs ±27% 0.53µs ± 1% -59.03% (p=0.008 n=5+5)
name old alloc/op new alloc/op delta
TBHelper-4 216B ± 0% 0B -100.00% (p=0.008 n=5+5)
name old allocs/op new allocs/op delta
TBHelper-4 2.00 ± 0% 0.00 -100.00% (p=0.008 n=5+5)
Change-Id: I6565feb491513815e1058637d086b0374fa94e19
GitHub-Last-Rev: c2329cf225
GitHub-Pull-Request: golang/go#38834
Reviewed-on: https://go-review.googlesource.com/c/go/+/231717
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Trust: Emmanuel Odeke <emmanuel@orijtech.com>
Currently, the benchmark output from the testing package prints small
values with three significant figures. This means it can only
distinguish 1 part in 100, or a 1% error, which can be enough to throw
off further analysis of the output. This CL increases it to four
significant figures. For time values, at least, anything beyond four
significant figures is almost certainly noise.
Fixes#34626.
Change-Id: I3bcf305427130026276e6a4c78167989319f280c
Reviewed-on: https://go-review.googlesource.com/c/go/+/267102
Trust: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
According to #28614, proposal review agreed in December 2018 that
Match should return an error for failed matches where the unmatched
part of the pattern has a syntax error. (The failed match has to date
caused the scan of the pattern to stop early.)
This change implements that behavior: the match loop continues
scanning to the end of the pattern, even after a confirmed mismatch,
to check whether the pattern is even well-formed.
The change applies to both path.Match and filepath.Match.
Then filepath.Glob and fs.Glob make a single validity-checking
call to Match before beginning their usual processing.
Also update fstest.TestFS to check for correct validation in custom
Glob implementations.
Fixes#28614.
Change-Id: Ic1d35a4bb9c3565184ae83dbefc425c5c96318e7
Reviewed-on: https://go-review.googlesource.com/c/go/+/264397
Trust: Russ Cox <rsc@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Supports 386 and amd64 architectures on all operating systems.
Example output:
$ go test -bench=.*
goos: darwin
goarch: amd64
pkg: strconv
cpu: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
BenchmarkAtof64Decimal-4 24431032 46.8 ns/op
...
As the displayed CPU information is only used for information
purposes it is lazily initialized when needed using the new
internal/sysinfo package.
This allows internal/cpu to stay without dependencies and avoid
initialization costs when the CPU information is not needed as
the new code to query the CPU name in internal/cpu can be
dead code eliminated if not used.
Fixes#39214
Change-Id: I77ae5c5d2fed6b28fa78dd45075f9f0a6a7f1bfd
Reviewed-on: https://go-review.googlesource.com/c/go/+/263804
Trust: Martin Möhrmann <moehrmann@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
The old ioutil references are still valid, but update our code
to reflect best practices and get used to the new locations.
Code compiled with the bootstrap toolchain
(cmd/asm, cmd/dist, cmd/compile, debug/elf)
must remain Go 1.4-compatible and is excluded.
Also excluded vendored code.
For #41190.
Change-Id: I6d86f2bf7bc37a9d904b6cee3fe0c7af6d94d5b1
Reviewed-on: https://go-review.googlesource.com/c/go/+/263142
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Add Stat helper function, StatFS interface, and test.
Add Stat method to fstest.MapFS.
Add testing of Stat method to fstest.TestFS.
For #41190.
Change-Id: Icf8b6eb1c3fa6f93a9be8405ec5a9468fb1da97b
Reviewed-on: https://go-review.googlesource.com/c/go/+/243913
Trust: Russ Cox <rsc@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
This change adds basic test helpers for file system code.
The type MapFS is a simple map-based file system for use when
exercising general file system code.
The func TestFS tests a file system implementation.
For #41190.
Change-Id: I5a2036f57e733915ad508651ad7317749794423c
Reviewed-on: https://go-review.googlesource.com/c/go/+/243910
Trust: Russ Cox <rsc@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
There are many reader behaviors that are subtle and
worth testing, and it's nice to have one complete tester
instead of many incomplete ones.
For #41190, which will use this as part of a larger
file system implementation tester.
Change-Id: Ib4cc7fae94b0d9b45dfacadc52baa77ad3761322
Reviewed-on: https://go-review.googlesource.com/c/go/+/243909
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Adds a benchmark for TB.Helper, to use as a judge of future
improvements like CL 231717.
Change-Id: I17c40d482fc12caa3eb2c1cda39fd8c42356b422
Reviewed-on: https://go-review.googlesource.com/c/go/+/257317
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
If a signal is sent to t.signal before the panic is triggered,
a panicking test may end up with "warning: no tests to run" because
the tRunner that invokes the test in t.Run calls runtime.Goexit on
panic, which causes the panicking test not be recorded in runTests.
Send the signal if and only if there is no panic.
Fixes#41479
Change-Id: I812f1303bfe02c443a1902732e68d21620d6672e
Reviewed-on: https://go-review.googlesource.com/c/go/+/256098
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
Trust: Bryan C. Mills <bcmills@google.com>
Previously, it was impossible to call FailNow in a Cleanup.
Because it can terminate a panicking goroutine and cause its
parent hangs on t.signal channel. This CL sends the signal
in a deferred call to prevent the hang.
Fixes#41355
Change-Id: I4552d3a7ea763ef86817bf9b50c0e37fb34bf20f
Reviewed-on: https://go-review.googlesource.com/c/go/+/254637
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Trust: Emmanuel Odeke <emm.odeke@gmail.com>
Ensures that calling TempDir() in either of Cleanup or Benchmark
doesn't cause test failures which were previously caused by the
created directory having been deleted after the first run, yet
we prevented the recreation of the directory due to our selection
of concurrency primitive sync.Once. This change recreates the
temporary directory if it doesn't exist, regardless of how
many times Cleanup and Benchmark are invoked.
Fixes#41062
Change-Id: I925d9f7207d7c369a193d1e17da7a59a586244a7
Reviewed-on: https://go-review.googlesource.com/c/go/+/251297
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
While debugging #40771, I realized that the chatty printer should only
ever print to a single io.Writer (normally os.Stdout). The other
Writer implementations in the chain write to local buffers, but if we
wrote a test's output to a local buffer, then we did *not* write it to
stdout and we should not store it as the most recently logged test.
Because the chatty printer should only ever print to one place, it
shouldn't receive an io.Writer as an argument — rather, it shouldn't
be used at all for destinations other than the main output stream.
On the other hand, when we flush the output buffer to stdout in the
top-level flushToParent call, it is important that we not allow some
other test's output to intrude between the test summary header and the
remainder of the test's output. cmd/test2json doesn't know how to
parse such an intrusion, and it's confusing to humans too.
No test because I couldn't reproduce the user-reported error without
modifying the testing package. (This behavior seems to be very
sensitive to output size and/or goroutine scheduling.)
Fixes#40771
Updates #38458
Change-Id: Ic19bf1d535672b096ba1c8583a3b74aab6d6d766
Reviewed-on: https://go-review.googlesource.com/c/go/+/249026
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Previously, if an example test invoked runtime.Goexit, it would
pass yet hang until a timeout, while regular tests that invoke
runtime.Goexit do fail. This change removes that inconsistent
behavior and makes such example tests fail, and panic with an
indication of having invoked runtime.Goexit.
Fixes#41084
Change-Id: I0ffa152204f2b1580f4d5d6961ba1ce6b13fc022
Reviewed-on: https://go-review.googlesource.com/c/go/+/251857
Reviewed-by: Emmanuel Odeke <emm.odeke@gmail.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
cmd/go.TestScript/test_main_twice demonstrates a program that invokes
(*M).Run twice in a row. If we only restore os.Exit(0) in m.afterOnce,
we will fail to restore it after the second run and fail the test
process despite both runs passing.
Updates #29062
Updates #23129
Change-Id: Id22ec68f1708e4583c8dda14a8ba0efae7178b85
Reviewed-on: https://go-review.googlesource.com/c/go/+/251262
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This catches cases where a test calls code that calls os.Exit(0),
thereby skipping all subsequent tests.
Fixes#29062
Change-Id: If9478972f40189e27623557e7141469ca4234d89
Reviewed-on: https://go-review.googlesource.com/c/go/+/250977
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
We could instead fix cmd/test2json to treat PAUSE lines as *not*
changing the active test name, but that seems like it would be more
confusing to humans, and also wouldn't fix tools that parse output
using existing builds of cmd/test2json.
Fixes#40657
Change-Id: I937611778f5b1e7dd1d6e9f44424d7e725a589ed
Reviewed-on: https://go-review.googlesource.com/c/go/+/248727
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Jean de Klerk <deklerk@google.com>
Corrects ErrReader's signature to what was accepted in the approved
proposal, and also removes an exported ErrIO which wasn't part of
the proposal and is unnecessary.
The new signature allows users to customize their own errors.
While here, started examples, with ErrReader leading the way.
Updates #38781
Change-Id: Ia7f84721f11061343cfef8b1adc2b7b69bc3f43c
Reviewed-on: https://go-review.googlesource.com/c/go/+/248898
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Paschalis Tsilias
Julien Tant
Ian Lance Taylor
Michael Fraenkel
徐志伟
KimMachineGun
Carlo Alberto Ferraris
Bryan C. Mills
Tao Qingyun
Josh Bleecher Snyder
fanzha02
Alexey Vilenski
Russ Cox
Austin Clements
DrGo
Bryan Boreham
Martin Möhrmann
Emmanuel T Odeke
Changkun Ou
Michał Łowicki
Carlos Alexandro Becker
Katie Hockman
Jean de Klerk