mirror of https://github.com/golang/go.git
7 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
Ian Lance Taylor
|
b0b1d42db3 |
all: change from sort functions to slices functions where feasible
Doing this because the slices functions are slightly faster and slightly easier to use. It also removes one dependency layer. This CL does not change packages that are used during bootstrap, as the bootstrap compiler does not have the required slices functions. It does not change the go/scanner package because the ErrorList Len, Swap, and Less methods are part of the Go 1 API. Change-Id: If52899be791c829198e11d2408727720b91ebe8a Reviewed-on: https://go-review.googlesource.com/c/go/+/587655 LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Commit-Queue: Ian Lance Taylor <iant@google.com> Reviewed-by: Damien Neil <dneil@google.com> |
|
Russ Cox
|
4f1b0a44cb |
all: update to use os.ReadFile, os.WriteFile, os.CreateTemp, os.MkdirTemp
As part of #42026, these helpers from io/ioutil were moved to os. (ioutil.TempFile and TempDir became os.CreateTemp and MkdirTemp.) Update the Go tree to use the preferred names. As usual, code compiled with the Go 1.4 bootstrap toolchain and code vendored from other sources is excluded. ReadDir changes are in a separate CL, because they are not a simple search and replace. For #42026. Change-Id: If318df0216d57e95ea0c4093b89f65e5b0ababb3 Reviewed-on: https://go-review.googlesource.com/c/go/+/266365 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org> |
|
|
Russ Cox
|
c32140fa94 |
all: update to use filepath.WalkDir instead of filepath.Walk
Now that filepath.WalkDir is available, it is more efficient and should be used in place of filepath.Walk. Update the tree to reflect best practices. As usual, the code compiled with Go 1.4 during bootstrap is excluded. (In this CL, that's only cmd/dist.) For #42027. Change-Id: Ib0f7b1e43e50b789052f9835a63ced701d8c411c Reviewed-on: https://go-review.googlesource.com/c/go/+/267719 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Rob Pike <r@golang.org> |
|
|
Russ Cox
|
7bb721b938 |
all: update references to symbols moved from os to io/fs
The old os references are still valid, but update our code to reflect best practices and get used to the new locations. Code compiled with the bootstrap toolchain (cmd/asm, cmd/dist, cmd/compile, debug/elf) must remain Go 1.4-compatible and is excluded. For #41190. Change-Id: I8f9526977867c10a221e2f392f78d7dec073f1bd Reviewed-on: https://go-review.googlesource.com/c/go/+/243907 Trust: Russ Cox <rsc@golang.org> Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Rob Pike <r@golang.org> |
|
|
Russ Cox
|
6ca324f283 |
index/suffixarray: index 3-10X faster in half the memory
This CL changes the index/suffixarray construction algorithm from QSufSort to SAIS. For an N-byte input, QSufSort runs in O(N log N) time and requires an N-int temporary work space in addition to the N-int output. In contrast, SAIS runs in O(N) time and, for essentially all real inputs, is able to use the N-int output buffer as its temporary work space. (In pathological cases, SAIS must allocate a temporary work space of at most N/2 ints. There exist more complex variants that guarantee to avoid the work space in all cases, but they hardly seem worth the cost given how rare these pathological cases are.) The SAIS code therefore uses 50% of the memory across the board. It also runs 3-10X faster on real input text. This CL also adds more extensive algorithmic tests, including an exhaustive test over small inputs to catch corner case problems. name old speed new speed delta New/text=opticks/size=100K/bits=32-12 6.15MB/s ± 1% 26.79MB/s ± 1% +335.89% (p=0.008 n=5+5) New/text=opticks/size=100K/bits=64-12 5.90MB/s ± 2% 27.29MB/s ± 2% +362.23% (p=0.008 n=5+5) New/text=opticks/size=500K/bits=32-12 4.99MB/s ± 3% 25.37MB/s ± 2% +408.01% (p=0.008 n=5+5) New/text=opticks/size=500K/bits=64-12 4.88MB/s ± 1% 25.66MB/s ± 4% +425.52% (p=0.008 n=5+5) New/text=go/size=100K/bits=32-12 5.81MB/s ± 1% 26.49MB/s ± 2% +355.85% (p=0.008 n=5+5) New/text=go/size=100K/bits=64-12 5.76MB/s ± 2% 26.65MB/s ± 3% +362.60% (p=0.008 n=5+5) New/text=go/size=500K/bits=32-12 4.91MB/s ± 1% 25.12MB/s ± 2% +411.86% (p=0.008 n=5+5) New/text=go/size=500K/bits=64-12 4.83MB/s ± 2% 25.79MB/s ± 2% +434.44% (p=0.008 n=5+5) New/text=go/size=1M/bits=32-12 4.62MB/s ± 2% 24.87MB/s ± 2% +438.78% (p=0.008 n=5+5) New/text=go/size=1M/bits=64-12 4.39MB/s ± 2% 24.61MB/s ± 2% +460.68% (p=0.008 n=5+5) New/text=go/size=5M/bits=32-12 2.85MB/s ± 2% 24.78MB/s ± 7% +768.33% (p=0.008 n=5+5) New/text=go/size=5M/bits=64-12 2.28MB/s ± 1% 18.70MB/s ± 7% +719.63% (p=0.008 n=5+5) New/text=go/size=10M/bits=32-12 2.08MB/s ± 1% 21.04MB/s ± 6% +909.60% (p=0.008 n=5+5) New/text=go/size=10M/bits=64-12 1.83MB/s ± 1% 16.64MB/s ± 2% +809.18% (p=0.008 n=5+5) New/text=go/size=50M/bits=32-12 1.51MB/s ± 0% 10.58MB/s ± 1% +602.52% (p=0.008 n=5+5) New/text=go/size=50M/bits=64-12 1.34MB/s ± 4% 9.00MB/s ± 1% +569.35% (p=0.008 n=5+5) New/text=zero/size=100K/bits=32-12 4.17MB/s ± 0% 157.56MB/s ± 1% +3678.42% (p=0.016 n=4+5) New/text=zero/size=100K/bits=64-12 4.19MB/s ± 2% 162.72MB/s ± 2% +3783.63% (p=0.008 n=5+5) New/text=zero/size=500K/bits=32-12 3.72MB/s ± 5% 159.17MB/s ± 1% +4176.57% (p=0.008 n=5+5) New/text=zero/size=500K/bits=64-12 3.77MB/s ± 3% 164.95MB/s ± 4% +4277.60% (p=0.008 n=5+5) New/text=zero/size=1M/bits=32-12 3.46MB/s ± 3% 158.42MB/s ± 1% +4476.08% (p=0.008 n=5+5) New/text=zero/size=1M/bits=64-12 3.41MB/s ± 4% 163.70MB/s ± 2% +4700.65% (p=0.008 n=5+5) New/text=zero/size=5M/bits=32-12 3.12MB/s ± 2% 151.92MB/s ± 4% +4775.48% (p=0.008 n=5+5) New/text=zero/size=5M/bits=64-12 3.09MB/s ± 2% 166.19MB/s ± 2% +5274.84% (p=0.008 n=5+5) New/text=zero/size=10M/bits=32-12 2.97MB/s ± 1% 157.75MB/s ± 1% +5211.38% (p=0.008 n=5+5) New/text=zero/size=10M/bits=64-12 2.92MB/s ± 1% 162.75MB/s ± 2% +5473.77% (p=0.008 n=5+5) New/text=zero/size=50M/bits=32-12 2.67MB/s ± 1% 144.43MB/s ± 5% +5305.39% (p=0.008 n=5+5) New/text=zero/size=50M/bits=64-12 2.61MB/s ± 1% 125.19MB/s ± 2% +4700.33% (p=0.016 n=5+4) New/text=rand/size=100K/bits=32-12 8.69MB/s ± 6% 27.60MB/s ± 1% +217.73% (p=0.008 n=5+5) New/text=rand/size=100K/bits=64-12 8.92MB/s ± 1% 26.37MB/s ± 4% +195.50% (p=0.008 n=5+5) New/text=rand/size=500K/bits=32-12 7.11MB/s ± 2% 25.23MB/s ± 2% +254.78% (p=0.008 n=5+5) New/text=rand/size=500K/bits=64-12 7.08MB/s ± 1% 25.45MB/s ± 2% +259.56% (p=0.008 n=5+5) New/text=rand/size=1M/bits=32-12 6.45MB/s ± 2% 24.47MB/s ± 3% +279.11% (p=0.008 n=5+5) New/text=rand/size=1M/bits=64-12 6.09MB/s ± 4% 23.00MB/s ± 4% +277.85% (p=0.008 n=5+5) New/text=rand/size=5M/bits=32-12 3.68MB/s ± 3% 10.34MB/s ± 5% +181.08% (p=0.008 n=5+5) New/text=rand/size=5M/bits=64-12 3.25MB/s ± 1% 6.23MB/s ± 1% +91.93% (p=0.008 n=5+5) New/text=rand/size=10M/bits=32-12 3.03MB/s ± 1% 5.61MB/s ± 2% +85.28% (p=0.008 n=5+5) New/text=rand/size=10M/bits=64-12 2.80MB/s ± 1% 4.29MB/s ± 2% +53.40% (p=0.008 n=5+5) New/text=rand/size=50M/bits=32-12 2.11MB/s ± 0% 2.45MB/s ± 1% +16.23% (p=0.029 n=4+4) New/text=rand/size=50M/bits=64-12 2.04MB/s ± 1% 2.24MB/s ± 1% +10.03% (p=0.016 n=5+4) SaveRestore/bits=32-12 327MB/s ± 5% 319MB/s ± 2% ~ (p=0.310 n=5+5) SaveRestore/bits=64-12 306MB/s ± 3% 306MB/s ± 2% ~ (p=0.841 n=5+5) name old alloc/op new alloc/op delta New/text=opticks/size=100K/bits=32-12 811kB ± 0% 401kB ± 0% -50.51% (p=0.008 n=5+5) New/text=opticks/size=100K/bits=64-12 1.62MB ± 0% 0.80MB ± 0% -50.51% (p=0.008 n=5+5) New/text=opticks/size=500K/bits=32-12 4.04MB ± 0% 2.01MB ± 0% -50.37% (p=0.008 n=5+5) New/text=opticks/size=500K/bits=64-12 8.07MB ± 0% 4.01MB ± 0% -50.36% (p=0.016 n=4+5) New/text=go/size=100K/bits=32-12 811kB ± 0% 401kB ± 0% ~ (p=0.079 n=4+5) New/text=go/size=100K/bits=64-12 1.62MB ± 0% 0.80MB ± 0% -50.50% (p=0.008 n=5+5) New/text=go/size=500K/bits=32-12 4.04MB ± 0% 2.01MB ± 0% ~ (p=0.079 n=4+5) New/text=go/size=500K/bits=64-12 8.07MB ± 0% 4.01MB ± 0% -50.36% (p=0.000 n=4+5) New/text=go/size=1M/bits=32-12 8.07MB ± 0% 4.01MB ± 0% -50.36% (p=0.008 n=5+5) New/text=go/size=1M/bits=64-12 16.1MB ± 0% 8.0MB ± 0% -50.36% (p=0.008 n=5+5) New/text=go/size=5M/bits=32-12 40.2MB ± 0% 20.0MB ± 0% -50.18% (p=0.008 n=5+5) New/text=go/size=5M/bits=64-12 80.3MB ± 0% 40.0MB ± 0% -50.18% (p=0.008 n=5+5) New/text=go/size=10M/bits=32-12 80.2MB ± 0% 40.0MB ± 0% -50.09% (p=0.000 n=5+4) New/text=go/size=10M/bits=64-12 160MB ± 0% 80MB ± 0% -50.09% (p=0.000 n=5+4) New/text=go/size=50M/bits=32-12 402MB ± 0% 200MB ± 0% -50.29% (p=0.000 n=5+4) New/text=go/size=50M/bits=64-12 805MB ± 0% 400MB ± 0% -50.29% (p=0.000 n=5+4) New/text=zero/size=100K/bits=32-12 1.46MB ± 0% 0.40MB ± 0% -72.46% (p=0.008 n=5+5) New/text=zero/size=100K/bits=64-12 3.02MB ± 0% 0.80MB ± 0% -73.45% (p=0.008 n=5+5) New/text=zero/size=500K/bits=32-12 8.66MB ± 0% 2.01MB ± 0% ~ (p=0.079 n=4+5) New/text=zero/size=500K/bits=64-12 19.7MB ± 0% 4.0MB ± 0% -79.63% (p=0.008 n=5+5) New/text=zero/size=1M/bits=32-12 19.7MB ± 0% 4.0MB ± 0% ~ (p=0.079 n=4+5) New/text=zero/size=1M/bits=64-12 39.0MB ± 0% 8.0MB ± 0% -79.48% (p=0.000 n=5+4) New/text=zero/size=5M/bits=32-12 85.2MB ± 0% 20.0MB ± 0% -76.52% (p=0.008 n=5+5) New/text=zero/size=5M/bits=64-12 169MB ± 0% 40MB ± 0% -76.27% (p=0.008 n=5+5) New/text=zero/size=10M/bits=32-12 169MB ± 0% 40MB ± 0% -76.26% (p=0.000 n=5+4) New/text=zero/size=10M/bits=64-12 333MB ± 0% 80MB ± 0% -75.99% (p=0.008 n=5+5) New/text=zero/size=50M/bits=32-12 739MB ± 0% 200MB ± 0% -72.93% (p=0.000 n=4+5) New/text=zero/size=50M/bits=64-12 1.63GB ± 0% 0.40GB ± 0% -75.42% (p=0.008 n=5+5) New/text=rand/size=100K/bits=32-12 807kB ± 0% 401kB ± 0% -50.25% (p=0.008 n=5+5) New/text=rand/size=100K/bits=64-12 1.61MB ± 0% 0.80MB ± 0% -50.25% (p=0.008 n=5+5) New/text=rand/size=500K/bits=32-12 4.04MB ± 0% 2.01MB ± 0% ~ (p=0.079 n=4+5) New/text=rand/size=500K/bits=64-12 8.07MB ± 0% 4.01MB ± 0% ~ (p=0.079 n=4+5) New/text=rand/size=1M/bits=32-12 8.07MB ± 0% 4.01MB ± 0% -50.36% (p=0.000 n=5+4) New/text=rand/size=1M/bits=64-12 16.1MB ± 0% 8.0MB ± 0% -50.36% (p=0.008 n=5+5) New/text=rand/size=5M/bits=32-12 40.3MB ± 0% 20.0MB ± 0% -50.35% (p=0.029 n=4+4) New/text=rand/size=5M/bits=64-12 80.7MB ± 0% 40.0MB ± 0% ~ (p=0.079 n=4+5) New/text=rand/size=10M/bits=32-12 80.7MB ± 0% 40.0MB ± 0% -50.41% (p=0.008 n=5+5) New/text=rand/size=10M/bits=64-12 161MB ± 0% 80MB ± 0% -50.44% (p=0.029 n=4+4) New/text=rand/size=50M/bits=32-12 403MB ± 0% 200MB ± 0% -50.36% (p=0.000 n=5+4) New/text=rand/size=50M/bits=64-12 806MB ± 0% 400MB ± 0% ~ (p=0.079 n=4+5) SaveRestore/bits=32-12 5.28MB ± 0% 5.28MB ± 0% ~ (p=1.000 n=5+5) SaveRestore/bits=64-12 9.47MB ± 0% 9.47MB ± 0% ~ (p=0.286 n=5+5) https://perf.golang.org/search?q=upload:20190426.1 Fixes #15480. Change-Id: I0790f6edf67f5a9c02b4462632b4942e0c37988b Reviewed-on: https://go-review.googlesource.com/c/go/+/174100 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Eric Roshan-Eisner <edre@google.com> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> |
|
|
Russ Cox
|
45be3530a3 |
index/suffixarray: add 32-bit implementation
The original index/suffixarray used 32-bit ints on 64-bit machines, because that's what 'int' meant in Go at the time. When we changed the meaning of int, that doubled the space overhead of suffix arrays for all uses, even though the vast majority of them describe less than 2 GB of text. The space overhead of a suffix array compared to the text is not insignificant: there's a big difference for many uses between 4X and 8X. This CL adjusts names in qsufsort.go so that a global search and replace s/32/64/g produces a working 64-bit implementation, and then it modifies suffixarray.go to choose between the 32-bit and 64-bit implementation as appropriate depending on the input size. The 64-bit implementation is generated by 'go generate'. This CL also restructures the benchmarks, to test different input sizes, different input texts, and 32-bit vs 64-bit. The serialized form uses varint-encoded numbers and is unchanged, so on-disk suffix arrays written by older versions of Go will be readable by this version, and vice versa. The 32-bit version runs a up to 17% faster than the 64-bit version on real inputs, but more importantly it uses 50% less memory. I have a followup CL that also implements a faster algorithm on top of these improvements, but these are a good first step. name 64-bit speed 32-bit speed delta New/text=opticks/size=100K/bits=*-12 4.44MB/s ± 0% 4.64MB/s ± 0% +4.41% (p=0.008 n=5+5) New/text=opticks/size=500K/bits=*-12 3.70MB/s ± 1% 3.82MB/s ± 0% +3.30% (p=0.008 n=5+5) New/text=go/size=100K/bits=*-12 4.40MB/s ± 0% 4.61MB/s ± 0% +4.82% (p=0.008 n=5+5) New/text=go/size=500K/bits=*-12 3.66MB/s ± 0% 3.77MB/s ± 0% +3.01% (p=0.016 n=4+5) New/text=go/size=1M/bits=*-12 3.29MB/s ± 0% 3.55MB/s ± 0% +7.90% (p=0.016 n=5+4) New/text=go/size=5M/bits=*-12 2.25MB/s ± 1% 2.65MB/s ± 0% +17.81% (p=0.008 n=5+5) New/text=go/size=10M/bits=*-12 1.82MB/s ± 0% 2.09MB/s ± 1% +14.36% (p=0.008 n=5+5) New/text=go/size=50M/bits=*-12 1.35MB/s ± 0% 1.51MB/s ± 1% +12.33% (p=0.008 n=5+5) New/text=zero/size=100K/bits=*-12 3.42MB/s ± 0% 3.32MB/s ± 0% -2.74% (p=0.000 n=5+4) New/text=zero/size=500K/bits=*-12 3.00MB/s ± 1% 2.97MB/s ± 0% -1.13% (p=0.016 n=5+4) New/text=zero/size=1M/bits=*-12 2.81MB/s ± 0% 2.78MB/s ± 2% ~ (p=0.167 n=5+5) New/text=zero/size=5M/bits=*-12 2.46MB/s ± 0% 2.53MB/s ± 0% +3.18% (p=0.008 n=5+5) New/text=zero/size=10M/bits=*-12 2.35MB/s ± 0% 2.42MB/s ± 0% +2.98% (p=0.016 n=4+5) New/text=zero/size=50M/bits=*-12 2.12MB/s ± 0% 2.18MB/s ± 0% +3.02% (p=0.008 n=5+5) New/text=rand/size=100K/bits=*-12 6.98MB/s ± 0% 7.22MB/s ± 0% +3.38% (p=0.016 n=4+5) New/text=rand/size=500K/bits=*-12 5.53MB/s ± 0% 5.64MB/s ± 0% +1.92% (p=0.008 n=5+5) New/text=rand/size=1M/bits=*-12 4.62MB/s ± 1% 5.06MB/s ± 0% +9.61% (p=0.008 n=5+5) New/text=rand/size=5M/bits=*-12 3.09MB/s ± 0% 3.43MB/s ± 0% +10.94% (p=0.016 n=4+5) New/text=rand/size=10M/bits=*-12 2.68MB/s ± 0% 2.95MB/s ± 0% +10.39% (p=0.008 n=5+5) New/text=rand/size=50M/bits=*-12 1.92MB/s ± 0% 2.06MB/s ± 1% +7.41% (p=0.008 n=5+5) SaveRestore/bits=*-12 243MB/s ± 1% 259MB/s ± 0% +6.68% (p=0.000 n=9+10) name 64-bit alloc/op 32-bit alloc/op delta New/text=opticks/size=100K/bits=*-12 1.62MB ± 0% 0.81MB ± 0% -50.00% (p=0.000 n=5+4) New/text=opticks/size=500K/bits=*-12 8.07MB ± 0% 4.04MB ± 0% -49.89% (p=0.008 n=5+5) New/text=go/size=100K/bits=*-12 1.62MB ± 0% 0.81MB ± 0% -50.00% (p=0.008 n=5+5) New/text=go/size=500K/bits=*-12 8.07MB ± 0% 4.04MB ± 0% -49.89% (p=0.029 n=4+4) New/text=go/size=1M/bits=*-12 16.1MB ± 0% 8.1MB ± 0% -49.95% (p=0.008 n=5+5) New/text=go/size=5M/bits=*-12 80.3MB ± 0% 40.2MB ± 0% ~ (p=0.079 n=4+5) New/text=go/size=10M/bits=*-12 160MB ± 0% 80MB ± 0% -50.00% (p=0.008 n=5+5) New/text=go/size=50M/bits=*-12 805MB ± 0% 402MB ± 0% -50.06% (p=0.029 n=4+4) New/text=zero/size=100K/bits=*-12 3.02MB ± 0% 1.46MB ± 0% ~ (p=0.079 n=4+5) New/text=zero/size=500K/bits=*-12 19.7MB ± 0% 8.7MB ± 0% -55.98% (p=0.008 n=5+5) New/text=zero/size=1M/bits=*-12 39.0MB ± 0% 19.7MB ± 0% -49.60% (p=0.000 n=5+4) New/text=zero/size=5M/bits=*-12 169MB ± 0% 85MB ± 0% -49.46% (p=0.029 n=4+4) New/text=zero/size=10M/bits=*-12 333MB ± 0% 169MB ± 0% -49.43% (p=0.000 n=5+4) New/text=zero/size=50M/bits=*-12 1.63GB ± 0% 0.74GB ± 0% -54.61% (p=0.008 n=5+5) New/text=rand/size=100K/bits=*-12 1.61MB ± 0% 0.81MB ± 0% -50.00% (p=0.000 n=5+4) New/text=rand/size=500K/bits=*-12 8.07MB ± 0% 4.04MB ± 0% -49.89% (p=0.000 n=5+4) New/text=rand/size=1M/bits=*-12 16.1MB ± 0% 8.1MB ± 0% -49.95% (p=0.029 n=4+4) New/text=rand/size=5M/bits=*-12 80.7MB ± 0% 40.3MB ± 0% -50.06% (p=0.008 n=5+5) New/text=rand/size=10M/bits=*-12 161MB ± 0% 81MB ± 0% -50.03% (p=0.008 n=5+5) New/text=rand/size=50M/bits=*-12 806MB ± 0% 403MB ± 0% -50.00% (p=0.016 n=4+5) SaveRestore/bits=*-12 9.47MB ± 0% 5.28MB ± 0% -44.29% (p=0.000 n=9+8) https://perf.golang.org/search?q=upload:20190126.1+|+bits:64+vs+bits:32 Fixes #6816. Change-Id: Ied2fbea519a202ecc43719debcd233344ce38847 Reviewed-on: https://go-review.googlesource.com/c/go/+/174097 Run-TryBot: Russ Cox <rsc@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> |
|
|
Russ Cox
|
c007ce824d |
build: move package sources from src/pkg to src
Preparation was in CL 134570043. This CL contains only the effect of 'hg mv src/pkg/* src'. For more about the move, see golang.org/s/go14nopkg. |