Some tests need to use os.Chdir, but the use is complicated because
- they must change back to the old working directory;
- they must not use t.Parallel.
Add Chdir that covers these cases, and sets PWD environment variable
to the new directory for the duration of the test for Unix platforms.
Unify the panic message when t.Parallel is used together with t.Setenv
or t.Chdir.
Add some tests.
For #62516.
Change-Id: Ib050d173b26eb28a27dba5a206b2d0d877d761c1
Reviewed-on: https://go-review.googlesource.com/c/go/+/529895
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Windows time.Now granularity is around 0.5ms on modern systems,
which introduces a significant noise into benchmark results.
Instead of relying time.Now use QueryPerformanceCounter, which
has significantly better granularity compared to time.Now.
│ TimeNow-32 │ HighPrecisionTimeNow-32 │
│ sec/op │ sec/op vs base │
4.812n ± 0% 30.580n ± 0% +535.43% (p=0.000 n=20)
Fixes#31160
Change-Id: Ib2a574d638c9c6762a2524212def02265574e267
Reviewed-on: https://go-review.googlesource.com/c/go/+/557315
Reviewed-by: Quim Muntal <quimmuntal@gmail.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Alex Brainman <alex.brainman@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
The original goal of rounding to readable b.N
was to make it easier to eyeball times.
However, proper analysis requires tooling
(such as benchstat) anyway.
Instead, take b.N as it comes.
This will reduce the impact of external noise
such as GC on benchmarks.
This requires reworking our iteration estimates.
We used to calculate the estimated ns/op
and then divide our target ns by that estimate.
However, this order of operations was destructive
when the ns/op was very small; rounding could
hide almost an order of magnitude of variation.
Instead, multiply first, then divide.
Also, make n an int64 to avoid overflow.
Prior to this change, we attempted to cap b.N at 1e9.
Due to rounding up, it was possible to get b.N as high as 2e9.
This change consistently enforces the 1e9 cap.
This change also reduces the wall time required to run benchmarks.
Here's the impact of this change on the wall time to run
all benchmarks once with benchtime=1s on some std packages:
name old time/op new time/op delta
bytes 306s ± 1% 238s ± 1% -22.24% (p=0.000 n=10+10)
encoding/json 112s ± 8% 99s ± 7% -11.64% (p=0.000 n=10+10)
net/http 54.7s ± 7% 44.9s ± 4% -17.94% (p=0.000 n=10+9)
runtime 957s ± 1% 714s ± 0% -25.38% (p=0.000 n=10+9)
strings 262s ± 1% 201s ± 1% -23.27% (p=0.000 n=10+10)
[Geo mean] 216s 172s -20.23%
Updates #24735
Change-Id: I7e38efb8e23c804046bf4fc065b3f5f3991d0a15
Reviewed-on: https://go-review.googlesource.com/c/go/+/112155
Reviewed-by: Austin Clements <austin@google.com>
This adds a ReportMetric method to testing.B that lets the user report
custom benchmark metrics and override built-in metrics.
Fixes#26037.
Change-Id: I8236fbde3683fc27bbe45cbbedfd377b435edf64
Reviewed-on: https://go-review.googlesource.com/c/go/+/166717
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Josh Bleecher Snyder <josharian@gmail.com>
Kir Kolyshkin
Egon Elbre
Josh Bleecher Snyder
Austin Clements
Russ Cox