Supports 386 and amd64 architectures on all operating systems.
Example output:
$ go test -bench=.*
goos: darwin
goarch: amd64
pkg: strconv
cpu: Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz
BenchmarkAtof64Decimal-4 24431032 46.8 ns/op
...
As the displayed CPU information is only used for information
purposes it is lazily initialized when needed using the new
internal/sysinfo package.
This allows internal/cpu to stay without dependencies and avoid
initialization costs when the CPU information is not needed as
the new code to query the CPU name in internal/cpu can be
dead code eliminated if not used.
Fixes#39214
Change-Id: I77ae5c5d2fed6b28fa78dd45075f9f0a6a7f1bfd
Reviewed-on: https://go-review.googlesource.com/c/go/+/263804
Trust: Martin Möhrmann <moehrmann@google.com>
Reviewed-by: Keith Randall <khr@golang.org>
While debugging #40771, I realized that the chatty printer should only
ever print to a single io.Writer (normally os.Stdout). The other
Writer implementations in the chain write to local buffers, but if we
wrote a test's output to a local buffer, then we did *not* write it to
stdout and we should not store it as the most recently logged test.
Because the chatty printer should only ever print to one place, it
shouldn't receive an io.Writer as an argument — rather, it shouldn't
be used at all for destinations other than the main output stream.
On the other hand, when we flush the output buffer to stdout in the
top-level flushToParent call, it is important that we not allow some
other test's output to intrude between the test summary header and the
remainder of the test's output. cmd/test2json doesn't know how to
parse such an intrusion, and it's confusing to humans too.
No test because I couldn't reproduce the user-reported error without
modifying the testing package. (This behavior seems to be very
sensitive to output size and/or goroutine scheduling.)
Fixes#40771
Updates #38458
Change-Id: Ic19bf1d535672b096ba1c8583a3b74aab6d6d766
Reviewed-on: https://go-review.googlesource.com/c/go/+/249026
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
In #24929, we decided to stream chatty test output. It looks like,
foo_test.go:138: TestFoo/sub-1: hello from subtest 1
foo_test.go:138: TestFoo/sub-2: hello from subtest 2
In this CL, we refactor the output to be grouped by === CONT lines, preserving
the old test-file-before-log-line behavior:
=== CONT TestFoo/sub-1
foo_test.go:138 hello from subtest 1
=== CONT TestFoo/sub-2
foo_test.go:138 hello from subtest 2
This should remove a layer of verbosity from tests, and make it easier to group
together related lines. It also returns to a more familiar format (the
pre-streaming format), whilst still preserving the streaming feature.
Fixes#38458
Change-Id: Iaef94c580d69cdd541b2ef055aa004f50d72d078
Reviewed-on: https://go-review.googlesource.com/c/go/+/229085
Run-TryBot: Emmanuel Odeke <emm.odeke@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Andrew Bonventre <andybons@golang.org>
The doc for testing.T and testing.B said that both test and benchmark
logs are printed to stderr, but in reality that wasn't the case.
CL 24311 fixed the doc for T, this change fixes it for B.
Fixes#36257
Change-Id: I0ff77ff44608f60320a1565b371c81e96039e71c
Reviewed-on: https://go-review.googlesource.com/c/go/+/212457
Reviewed-by: Gabriel Aszalos <gabriel.aszalos@gmail.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Gabriel Aszalos <gabriel.aszalos@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Fixes#24929
Change-Id: Icc426068cd73b75b78001f55e1e5d81ccebbe854
Reviewed-on: https://go-review.googlesource.com/c/go/+/127120
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Any code that imports the testing package forces the testing flags to be
defined, even in non-test binaries. People work around this today by
defining a copy of the testing.TB interface just to avoid importing
testing.
Fix this by moving flag registration into a new function, testing.Init.
Delay calling Init until the testing binary begins to run, in
testing.MainStart.
Init is exported for cases where users need the testing flags to be
defined outside of a "go test" context. In particular, this may be
needed where testing.Benchmark is called outside of a test.
Fixes#21051
Change-Id: Ib7e02459e693c26ae1ba71bbae7d455a91118ee3
Reviewed-on: https://go-review.googlesource.com/c/go/+/173722
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
CL 166717 changed the way ns/op benchmark results were printed and
inadvertently rounded all ns/op results down to an integer, even if
they were small enough to print with digits after the decimal place.
For example, prior to this change, we got output like
BenchmarkFast-8 380491575 3.12 ns/op
CL 166717 changed this to
BenchmarkFast-8 380491575 3.00 ns/op
This had the further side-effect that ns/op values between 0 and 1
would not be printed at all because they would be rounded down to 0.
This CL fixes this by always recomputing the float64 value of ns/op
instead of using the int64 truncation from BenchmarkResult.NsPerOp.
Fixes#30997. Fixes#31005.
Change-Id: I21f73b9d5cc5ad41e7ff535675d07ca00051ecd7
Reviewed-on: https://go-review.googlesource.com/c/go/+/168937
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
The original goal of rounding to readable b.N
was to make it easier to eyeball times.
However, proper analysis requires tooling
(such as benchstat) anyway.
Instead, take b.N as it comes.
This will reduce the impact of external noise
such as GC on benchmarks.
This requires reworking our iteration estimates.
We used to calculate the estimated ns/op
and then divide our target ns by that estimate.
However, this order of operations was destructive
when the ns/op was very small; rounding could
hide almost an order of magnitude of variation.
Instead, multiply first, then divide.
Also, make n an int64 to avoid overflow.
Prior to this change, we attempted to cap b.N at 1e9.
Due to rounding up, it was possible to get b.N as high as 2e9.
This change consistently enforces the 1e9 cap.
This change also reduces the wall time required to run benchmarks.
Here's the impact of this change on the wall time to run
all benchmarks once with benchtime=1s on some std packages:
name old time/op new time/op delta
bytes 306s ± 1% 238s ± 1% -22.24% (p=0.000 n=10+10)
encoding/json 112s ± 8% 99s ± 7% -11.64% (p=0.000 n=10+10)
net/http 54.7s ± 7% 44.9s ± 4% -17.94% (p=0.000 n=10+9)
runtime 957s ± 1% 714s ± 0% -25.38% (p=0.000 n=10+9)
strings 262s ± 1% 201s ± 1% -23.27% (p=0.000 n=10+10)
[Geo mean] 216s 172s -20.23%
Updates #24735
Change-Id: I7e38efb8e23c804046bf4fc065b3f5f3991d0a15
Reviewed-on: https://go-review.googlesource.com/c/go/+/112155
Reviewed-by: Austin Clements <austin@google.com>
This adds a ReportMetric method to testing.B that lets the user report
custom benchmark metrics and override built-in metrics.
Fixes#26037.
Change-Id: I8236fbde3683fc27bbe45cbbedfd377b435edf64
Reviewed-on: https://go-review.googlesource.com/c/go/+/166717
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Fixes StartTimer's doc with the verb 'be'
that was previously missing in 'can also used'.
Change-Id: I4b3e6103fbf62d676056d32fcce4618536b7c05c
Reviewed-on: https://go-review.googlesource.com/c/go/+/165117
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
When running benchmarks with profilers and trying to
compare one run against another, it is very useful to be
able to force each run to execute exactly the same number
of iterations.
Discussion on the proposal issue #24735 led to the decision
to overload -benchtime, so that instead of saying
-benchtime 10s to run a benchmark for 10 seconds,
you say -benchtime 100x to run a benchmark 100 times.
Fixes#24735.
Change-Id: Id17c5bd18bd09987bb48ed12420d61ae9e200fd7
Reviewed-on: https://go-review.googlesource.com/c/139258
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
This reverts golang.org/cl/110775
Reason for revert: this is causing huge slow-dows on every run after
the 1st, on various benchmarks, on multiple architectures (see Issue
25622 for details). It's just a nice-to-have little optimization, and
we're near the 1st go1.11 beta release, so I'm reverting it.
Fixes#25622
Change-Id: I758ade4af4abf764abd8336d404396992d11a0c6
Reviewed-on: https://go-review.googlesource.com/115535
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Since subtests and subbenchmarks run in a separate goroutine, and thus
a separate stack, this entails capturing the stack trace at the point
tb.Run is called. The work of getting the file and line information from
this stack is only done when needed, however.
Continuing the search into the parent test also requires temporarily
holding its mutex. Since Run does not hold it while waiting for the
subtest to complete, there should be no risk of a deadlock due to this.
Fixes#24128
Change-Id: If0bb169f3ac96bd48794624e619ade7edb599f83
Reviewed-on: https://go-review.googlesource.com/108658
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
When running a benchmark multiple times, instead of re-computing the
value of b.N each time, use the value found by the first run.
For
go test -bench=. -benchtime 3s -count 2 p_test.go
on the benchmark in the linked issue; before:
BenchmarkBenchmark-4 500 10180593 ns/op
--- BENCH: BenchmarkBenchmark-4
p_test.go:13: single call took 10.111079ms
p_test.go:13: single call took 1.017298685s
p_test.go:13: single call took 5.090096124s
BenchmarkBenchmark-4 500 10182164 ns/op
--- BENCH: BenchmarkBenchmark-4
p_test.go:13: single call took 10.098169ms
p_test.go:13: single call took 1.017712905s
p_test.go:13: single call took 5.090898517s
PASS
ok command-line-arguments 12.244s
and after:
BenchmarkBenchmark-4 500 10177076 ns/op
--- BENCH: BenchmarkBenchmark-4
p_test.go:13: single call took 10.091301ms
p_test.go:13: single call took 1.016943125s
p_test.go:13: single call took 5.088376028s
BenchmarkBenchmark-4 500 10171497 ns/op
--- BENCH: BenchmarkBenchmark-4
p_test.go:13: single call took 10.140245ms
p_test.go:13: single call took 5.085605921s
PASS
ok command-line-arguments 11.218s
Fixes#23423
Change-Id: Ie66a8c5ac43881eb8741e14105db28745b4d56d3
Reviewed-on: https://go-review.googlesource.com/110775
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
It's not safe (it crashes), and it's also useless: if you run
multiple benchmarks in parallel you will not get reliable
timing results from any of them.
Fixes#18603.
Change-Id: I00e5a72f7c98151543cf7d5573c38383276e391a
Reviewed-on: https://go-review.googlesource.com/80841
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Most of these are return values that were part of a receiving parameter,
so they're still accessible.
A few others are not, but those have never had a use.
Found with github.com/mvdan/unparam, after Kevin Burke's suggestion that
the tool should also warn about unused result parameters.
Change-Id: Id8b5ed89912a99db22027703a88bd94d0b292b8b
Reviewed-on: https://go-review.googlesource.com/55910
Run-TryBot: Daniel Martí <mvdan@mvdan.cc>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Currently all package tests are executed once
with Parallel tests executed in parallel.
Then this process is repeated count*cpu times.
Tests are not parallelized over count*cpu.
Parallelizing over cpu is not possible as
GOMAXPROCS is a global setting. But it is
possible for count.
Parallelize over count.
Brings down testing of my package with -count=100
form 10s to 0.3s.
Change-Id: I76d8322adeb8c5c6e70b99af690291fd69d6402a
Reviewed-on: https://go-review.googlesource.com/44830
Run-TryBot: Dmitry Vyukov <dvyukov@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Now that ReadMemStats is fast (CL 34937), CL 36791 is not so
necessary, and causes confusion. See #20863
This was already partially reverted in CL 46612 but missed two of the
spots.
Fixes#20863
Change-Id: I1307a0f7b1f9e86e8b6ceaa6a677f24f13431110
Reviewed-on: https://go-review.googlesource.com/47350
Reviewed-by: Austin Clements <austin@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Apparently, "all such calls must happen" means that the t.Run call
must *return* before the outer test function returns, or the calls
will cause a data race on t.ran.
Clarify the docs.
Fixes#20339
Change-Id: I191a9af2a9095be1e0aaf10b79c30e00a9c495cb
Reviewed-on: https://go-review.googlesource.com/47150
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
If the only way the user indicates they want alloc stats shown
is via ReportAllocs, we don't know that until benchFunc is run.
Therefore, StopTimer's ReadMemStats will return incorrect data
for single cycle runs since there's no counterpart ReadMemStats from
StartTimer that initializes alloc stats.
It appears that this bug was introduced by CL 46612,
"testing: only call ReadMemStats if necessary when benchmarking"
Fixes#20590
Change-Id: I3b5ef91677823f4b98011880a3be15423baf7e33
Reviewed-on: https://go-review.googlesource.com/46612
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
If you have BenchmarkX1 with sub-benchmark Y
and you have BenchmarkX2 with no sub-benchmarks,
then
go test -bench=X/Y
runs BenchmarkX1 once with b.N=1 (to find out about Y)
and then not again, because it has sub-benchmarks,
but arguably also because we're interested in Y.
In contrast, it runs BenchmarkX2 in full, even though clearly
that is not relevant to the match X/Y. We do have to run X2
once with b.N=1 to probe for having X2/Y, but we should not
run it with larger b.N.
Fixes#20589.
Change-Id: Ib86907e844f34dcaac6cd05757f57db1019201d0
Reviewed-on: https://go-review.googlesource.com/46031
Run-TryBot: Russ Cox <rsc@golang.org>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
The run1 call removed in golang.org/cl/36990 was necessary to
initialize the duration of the benchmark. With it gone, the math in
launch() starts from 100. This doesn't work out well for second-long
benchmark methods. Put it back.
Updates #18815
Change-Id: I461f3466c805d0c61124a2974662f7ad45335794
Reviewed-on: https://go-review.googlesource.com/37530
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
When running benchmarks with -cpuprofile,
the entire process gets profiled,
and ReadMemStats is surprisingly expensive.
Running the sort benchmarks right now with
-cpuprofile shows almost half of all execution
time in ReadMemStats.
Since ReadMemStats is not required if the benchmark
does not need allocation stats, simply skip it.
This will make cpu profiles nicer to read
and significantly speed up the process of running benchmarks.
It might also make sense to toggle cpu profiling
on/off as we begin/end individual benchmarks,
but that wouldn't get us the time savings of
skipping ReadMemStats, so this CL is useful in itself.
Change-Id: I425197b1ee11be4bc91d22b929e2caf648ebd7c5
Reviewed-on: https://go-review.googlesource.com/36791
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
When running benchmarks, print "goos", "goarch", and "pkg"
labels. This makes it easier to refer to benchmark logs and understand
how they were generated. "pkg" is printed only for benchmarks located
in GOPATH.
Change-Id: I397cbdd57b9fe8cbabbb354ec7bfba59f5625c42
Reviewed-on: https://go-review.googlesource.com/36356
Run-TryBot: Quentin Smith <quentin@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
From the doc comment:
Context returns the context for the current test or benchmark.
The context is cancelled when the test or benchmark finishes.
A goroutine started during a test or benchmark can wait for the
context's Done channel to become readable as a signal that the
test or benchmark is over, so that the goroutine can exit.
Fixes#16221.
Fixes#17552.
Change-Id: I657df946be2c90048cc74615436c77c7d9d1226c
Reviewed-on: https://go-review.googlesource.com/31724
Reviewed-by: Rob Pike <r@golang.org>
For example, testing the current directory:
$ go test -run XXX
testing: warning: no tests to run
PASS
ok testing 0.013s
$
And in a summary:
$ go test -run XXX testing
ok testing 0.013s [no tests to run]
$
These make it easy to spot when the -run regexp hasn't matched anything
or there are no tests. Previously the message was printed in the "current directory"
case when there were no tests at all, but not for no matches, and either way
was not surfaced in the directory list summary form.
Fixes#15211.
Change-Id: I1c82a423d6bd429fb991c9ca964c9d26c96fd3c5
Reviewed-on: https://go-review.googlesource.com/22341
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
When ns/op dropped below 1, the old code
ignored benchtime and reverted to 1s.
Change-Id: I59752cef88d8d73bfd5b085f5400ae657f78504e
Reviewed-on: https://go-review.googlesource.com/26664
Run-TryBot: Josh Bleecher Snyder <josharian@gmail.com>
Reviewed-by: Marcel van Lohuizen <mpvl@golang.org>
This introduces a few changes
- Skipped benchmarks now print a SKIP line, also if there was
no output
- The benchmark name is only printed if there the benchmark
was not skipped or did not fail in the probe phase.
It also fixes a bug of doubling a skip message in chatty mode in
absense of a failure.
The chatty flag is now passed in the common struct to allow
for testing of the printed messages.
Fixes#14799
Change-Id: Ia8eb140c2e5bb467e66b8ef20a2f98f5d95415d5
Reviewed-on: https://go-review.googlesource.com/21504
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Change control flow to probe with N=1. This calls benchFunc
the same number of times as the old implementation in the
absence of subbenchmarks.
To be compatible with existing tools, benchmarking only
prints a line for "leaf" benchmarks. This means, though, that
the name of a benchmark can only be printed after the first
iteration.
Issue #14863
Change-Id: Ic7b9b89b058f8ebb5287755f24f9e47df8c9537c
Reviewed-on: https://go-review.googlesource.com/21043
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
The matcher is responsible for sanitizing and uniquing the
test and benchmark names and thus needs to be included before the
API can be exposed.
Matching currently uses the regexp to only match the top-level
tests/benchmarks.
Support for subtest matching is for another CL.
Change-Id: I7c8464068faef7ebc179b03a7fe3d01122cc4f0b
Reviewed-on: https://go-review.googlesource.com/18897
Reviewed-by: Russ Cox <rsc@golang.org>
Run-TryBot: Marcel van Lohuizen <mpvl@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
API is not exposed yet.
Change-Id: I729360ef2be1d8ea683ca93cdb1763897cc8657c
Reviewed-on: https://go-review.googlesource.com/18895
Reviewed-by: Russ Cox <rsc@golang.org>
testing.go:
- run method will evolve into the Run method.
- added level field in common
benchmark.go:
- benchContext will be central to distinguish handling of benchmarks
between normal Run methods and ones called from within Benchmark
function.
- expandCPU will evolve into the processing hook for Run methods
called within normal processing.
- runBench will evolve into the Run method.
Change-Id: I1816f9985d5ba94deb0ad062302ea9aee0bb5338
Reviewed-on: https://go-review.googlesource.com/18894
Reviewed-by: Russ Cox <rsc@golang.org>
The biggest change is that each test is now responsible for managing
the starting and stopping of its parallel subtests.
The "Main" test could be run as a tRunner as well. This shows that
the introduction of subtests is merely a generalization of and
consistent with the current semantics.
Change-Id: Ibf8388c08f85d4b2c0df69c069326762ed36a72e
Reviewed-on: https://go-review.googlesource.com/18893
Reviewed-by: Russ Cox <rsc@golang.org>
The tree's pretty inconsistent about single space vs double space
after a period in documentation. Make it consistently a single space,
per earlier decisions. This means contributors won't be confused by
misleading precedence.
This CL doesn't use go/doc to parse. It only addresses // comments.
It was generated with:
$ perl -i -npe 's,^(\s*// .+[a-z]\.) +([A-Z]),$1 $2,' $(git grep -l -E '^\s*//(.+\.) +([A-Z])')
$ go test go/doc -update
Change-Id: Iccdb99c37c797ef1f804a94b22ba5ee4b500c4f7
Reviewed-on: https://go-review.googlesource.com/20022
Reviewed-by: Rob Pike <r@golang.org>
Reviewed-by: Dave Day <djd@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Moves the implementation of RunBenchmarks to a non-exported function
that returns whether the execution was OK, and uses that to identify
failure in benchmarks.The exported function is kept for compatibility.
Like before, benchmarks will only be executed if tests and examples
pass. The PASS message will not be printed if there was a failure in
a benchmark.
Example output
BenchmarkThatCallsFatal-8 --- FAIL: BenchmarkThatCallsFatal-8
x_test.go:6: called by benchmark
FAIL
exit status 1
FAIL _/.../src/cmd/go/testdata/src/benchfatal 0.009s
Fixes#14307.
Change-Id: I6f3ddadc7da8a250763168cc099ae8b325a79602
Reviewed-on: https://go-review.googlesource.com/19889
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Martin Möhrmann
Bryan C. Mills
Jean de Klerk
Roger Peppe
Alberto Donizetti
vovapi
Caleb Spare
Austin Clements
Josh Bleecher Snyder
Emmanuel T Odeke
Tobias Klauser
Russ Cox
Diogo Pinela
Daniel Martí
Dmitry Vyukov
Brad Fitzpatrick
Meir Fischer
Heschi Kreinick
Marcel van Lohuizen
Quentin Smith
Joe Tsai
Caio Marcelo de Oliveira Filho