The Go programming language
Go to file
Tom Thorogood 9e7bc80b31 os: reuse readdir buffers on unix with a sync.Pool
(*File).readdir allocates a fixed-size 8KiB buffer on unix systems that
cannot be reused. While it accounts for just a single allocation, it's
more than large enough to show up in profiles and make things quite a
bit slower.

Instead of allocating so often, use a sync.Pool to allow these buffers to
be reused. This has a large impact on readdir heavy workloads.

Package os benchmarks:

name            old time/op    new time/op    delta
Readdirname-12    35.6µs ± 5%    18.1µs ± 4%  -49.00%  (p=0.000 n=10+10)
Readdir-12         142µs ± 1%     121µs ± 0%  -14.87%  (p=0.000 n=10+9)
ReadDir-12        44.0µs ± 6%    28.4µs ± 8%  -35.58%  (p=0.000 n=9+10)

name            old alloc/op   new alloc/op   delta
Readdirname-12    14.4kB ± 0%     6.2kB ± 0%  -57.08%  (p=0.000 n=10+10)
Readdir-12        41.6kB ± 0%    33.4kB ± 0%  -19.77%  (p=0.000 n=10+9)
ReadDir-12        21.9kB ± 0%    13.7kB ± 0%  -37.39%  (p=0.000 n=10+10)

name            old allocs/op  new allocs/op  delta
Readdirname-12       131 ± 0%       130 ± 0%   -0.76%  (p=0.000 n=10+10)
Readdir-12           367 ± 0%       366 ± 0%   -0.27%  (p=0.000 n=10+10)
ReadDir-12           249 ± 0%       248 ± 0%   -0.40%  (p=0.000 n=10+10)

A clunky benchmark I threw together that calls filepath.WalkDir on $GOMODCACHE:

name        old time/op    new time/op    delta
WalkDir-12    91.2ms ±19%    48.7ms ± 0%  -46.54%  (p=0.000 n=10+10)

name        old alloc/op   new alloc/op   delta
WalkDir-12    54.0MB ± 0%     7.6MB ± 0%  -85.92%  (p=0.000 n=8+9)

name        old allocs/op  new allocs/op  delta
WalkDir-12      136k ± 0%      130k ± 0%   -4.15%  (p=0.000 n=8+8)

Change-Id: I00e4d48726da0e46c528ab205409afd03127b844
Reviewed-on: https://go-review.googlesource.com/c/go/+/302169
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Run-TryBot: Emmanuel Odeke <emmanuel@orijtech.com>
TryBot-Result: Go Bot <gobot@golang.org>
2021-04-03 21:06:18 +00:00
.github .github: add link to questions in ISSUE_TEMPLATE 2020-01-06 17:05:31 +00:00
api syscall: restore broken GetQueuedCompletionStatus signature but make it not crash 2021-02-24 23:35:00 +00:00
doc cmd/go: assume Go 1.16 instead of Go 1.11 for dependencies that lack explicit 'go' directives 2021-03-19 19:59:56 +00:00
lib/time lib/time, time/tzdata: update tzdata to 2021a 2021-01-25 16:08:46 +00:00
misc cmd/compile, cmd/link: dynamically export writable static tmps 2021-03-16 23:14:54 +00:00
src os: reuse readdir buffers on unix with a sync.Pool 2021-04-03 21:06:18 +00:00
test cmd/compile: mark unused values as invalid to prevent problems in expandCalls 2021-04-02 17:58:59 +00:00
.gitattributes all: treat all files as binary, but check in .bat with CRLF 2020-06-08 15:31:43 +00:00
.gitignore
AUTHORS A+C: add new e-mail addresses for Andy Pan 2021-03-12 02:56:04 +00:00
CONTRIBUTING.md
CONTRIBUTORS A+C: add new e-mail addresses for Andy Pan 2021-03-12 02:56:04 +00:00
LICENSE
PATENTS
README.md README: pull gopher image from website 2021-02-16 18:25:10 +00:00
SECURITY.md
codereview.cfg codereview.cfg: add codereview.cfg for master branch 2021-02-19 18:44:53 +00:00

README.md

The Go Programming Language

Go is an open source programming language that makes it easy to build simple, reliable, and efficient software.

Gopher image Gopher image by Renee French, licensed under Creative Commons 3.0 Attributions license.

Our canonical Git repository is located at https://go.googlesource.com/go. There is a mirror of the repository at https://github.com/golang/go.

Unless otherwise noted, the Go source files are distributed under the BSD-style license found in the LICENSE file.

Download and Install

Binary Distributions

Official binary distributions are available at https://golang.org/dl/.

After downloading a binary release, visit https://golang.org/doc/install for installation instructions.

Install From Source

If a binary distribution is not available for your combination of operating system and architecture, visit https://golang.org/doc/install/source for source installation instructions.

Contributing

Go is the work of thousands of contributors. We appreciate your help!

To contribute, please read the contribution guidelines at https://golang.org/doc/contribute.html.

Note that the Go project uses the issue tracker for bug reports and proposals only. See https://golang.org/wiki/Questions for a list of places to ask questions about the Go language.