This reverts CL 525455. The test fails to build on darwin, alpine, and
android.
For #62440.
Change-Id: I39c6b1e16499bd61e0f166de6c6efe7a07961e62
Reviewed-on: https://go-review.googlesource.com/c/go/+/527317
Auto-Submit: Michael Pratt <mpratt@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Since CL 495855, Ms are cached for C threads calling into Go, including
the stack bounds of the system stack.
Some C libraries (e.g., coroutine libraries) do manual stack management
and may change stacks between calls to Go on the same thread.
Changing the stack if there is more Go up the stack would be
problematic. But if the calls are completely independent there is no
particular reason for Go to care about the changing stack boundary.
Thus, this CL allows the stack bounds to change in such cases. The
primary downside here (besides additional complexity) is that normal
systems that do not manipulate the stack may not notice unintentional
stack corruption as quickly as before.
Note that callbackUpdateSystemStack is written to be usable for the
initial setup in needm as well as updating the stack in cgocallbackg.
Fixes#62440.
For #62130.
Change-Id: I7841b056acea1111bdae3b718345a3bd3961b4a8
Reviewed-on: https://go-review.googlesource.com/c/go/+/525455
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
When passing pointers of Go objects from Go to C, the cgo command generate _Cgo_use(pN) for the unsafe.Pointer type arguments, so that the Go compiler will escape these object to heap.
Since the C function may callback to Go, then the Go stack might grow/shrink, that means the pointers that the C function have will be invalid.
After adding the #cgo noescape annotation for a C function, the cgo command won't generate _Cgo_use(pN), and the Go compiler won't force the object escape to heap.
After adding the #cgo nocallback annotation for a C function, which means the C function won't callback to Go, if it do callback to Go, the Go process will crash.
Fixes#56378
Change-Id: Ifdca070584e0d349c7b12276270e50089e481f7a
GitHub-Last-Rev: f1a17b08b0
GitHub-Pull-Request: golang/go#60399
Reviewed-on: https://go-review.googlesource.com/c/go/+/497837
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
The non-cgo test points Segv and TgkillSegv are currently in
testprogcgo. Although the test points don't explicitly use cgo,
being a cgo program, there is still some C code that runs when
the test point is invoked, such as thread creation code.
For the cgo test points, sometimes we fail to unwind the stack if
C code is involved. For the non-cgo ones, we want to always be
able to unwind the stack, so we check for stack unwinding failures.
But if a signal is landed in the small piece of C code mentioned
above, we may still fail to unwind. Move the non-cgo test points
to a pure-Go program to avoid this problem.
May fix#52963.
Updates #59029, #59443, #59492.
Change-Id: I35d99a0dd4c7cdb627e2083d2414887a24a2822d
Reviewed-on: https://go-review.googlesource.com/c/go/+/500535
Reviewed-by: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Cherry Mui <cherryyz@google.com>
It should be impossible for the program to exit with SIGCHLD,
but it happens occasionally. Skip the test on Darwin.
For #60316
Change-Id: Idc9d89838e73f077afc42a9703554d61ac7a0069
Reviewed-on: https://go-review.googlesource.com/c/go/+/497055
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
This reapplies CL 485500, with a fix drafted in CL 492987 incorporated.
CL 485500 is reverted due to #60004 and #60007. #60004 is fixed in
CL 492743. #60007 is fixed in CL 492987 (incorporated in this CL).
[Original CL 485500 description]
This reapplies CL 481061, with the followup fixes in CL 482975, CL 485315, and
CL 485316 incorporated.
CL 481061, by doujiang24 <doujiang24@gmail.com>, speed up C to Go
calls by binding the M to the C thread. See below for its
description.
CL 482975 is a followup fix to a C declaration in testprogcgo.
CL 485315 is a followup fix for x_cgo_getstackbound on Illumos.
CL 485316 is a followup cleanup for ppc64 assembly.
CL 479915 passed the G to _cgo_getstackbound for direct updates to
gp.stack.lo. A G can be reused on a new thread after the previous thread
exited. This could trigger the C TSAN race detector because it couldn't
see the synchronization in Go (lockextra) preventing the same G from
being used on multiple threads at the same time.
We work around this by passing the address of a stack variable to
_cgo_getstackbound rather than the G. The stack is generally unique per
thread, so TSAN won't see the same address from multiple threads. Even
if stacks are reused across threads by pthread, C TSAN should see the
synchonization in the stack allocator.
A regression test is added to misc/cgo/testsanitizer.
[Original CL 481061 description]
This reapplies CL 392854, with the followup fixes in CL 479255,
CL 479915, and CL 481057 incorporated.
CL 392854, by doujiang24 <doujiang24@gmail.com>, speed up C to Go
calls by binding the M to the C thread. See below for its
description.
CL 479255 is a followup fix for a small bug in ARM assembly code.
CL 479915 is another followup fix to address C to Go calls after
the C code uses some stack, but that CL is also buggy.
CL 481057, by Michael Knyszek, is a followup fix for a memory leak
bug of CL 479915.
[Original CL 392854 description]
In a C thread, it's necessary to acquire an extra M by using needm while invoking a Go function from C. But, needm and dropm are heavy costs due to the signal-related syscalls.
So, we change to not dropm while returning back to C, which means binding the extra M to the C thread until it exits, to avoid needm and dropm on each C to Go call.
Instead, we only dropm while the C thread exits, so the extra M won't leak.
When invoking a Go function from C:
Allocate a pthread variable using pthread_key_create, only once per shared object, and register a thread-exit-time destructor.
And store the g0 of the current m into the thread-specified value of the pthread key, only once per C thread, so that the destructor will put the extra M back onto the extra M list while the C thread exits.
When returning back to C:
Skip dropm in cgocallback, when the pthread variable has been created, so that the extra M will be reused the next time invoke a Go function from C.
This is purely a performance optimization. The old version, in which needm & dropm happen on each cgo call, is still correct too, and we have to keep the old version on systems with cgo but without pthreads, like Windows.
This optimization is significant, and the specific value depends on the OS system and CPU, but in general, it can be considered as 10x faster, for a simple Go function call from a C thread.
For the newly added BenchmarkCGoInCThread, some benchmark results:
1. it's 28x faster, from 3395 ns/op to 121 ns/op, in darwin OS & Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
2. it's 6.5x faster, from 1495 ns/op to 230 ns/op, in Linux OS & Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
[CL 479915 description]
Currently, when C calls into Go the first time, we grab an M
using needm, which sets m.g0's stack bounds using the SP. We don't
know how big the stack is, so we simply assume 32K. Previously,
when the Go function returns to C, we drop the M, and the next
time C calls into Go, we put a new stack bound on the g0 based on
the current SP. After CL 392854, we don't drop the M, and the next
time C calls into Go, we reuse the same g0, without recomputing
the stack bounds. If the C code uses quite a bit of stack space
before calling into Go, the SP may be well below the 32K stack
bound we assumed, so the runtime thinks the g0 stack overflows.
This CL makes needm get a more accurate stack bound from
pthread. (In some platforms this may still be a guess as we don't
know exactly where we are in the C stack), but it is probably
better than simply assuming 32K.
[CL 492987 description]
On the first call into Go from a C thread, currently we set the g0
stack's high bound imprecisely based on the SP. With CL 485500, we
keep the M and don't recompute the stack bounds when it calls into
Go again. If the first call is made when the C thread uses some
deep stack, but a subsequent call is made with a shallower stack,
the SP may be above g0.stack.hi.
This is usually okay as we don't check usually stack.hi. One place
where we do check for stack.hi is in the signal handler, in
adjustSignalStack. In particular, C TSAN delivers signals on the
g0 stack (instead of the usual signal stack). If the SP is above
g0.stack.hi, we don't see it is on the g0 stack, and throws.
This CL makes it get an accurate stack upper bound with the
pthread API (on the platforms where it is available).
Also add some debug print for the "handler not on signal stack"
throw.
Fixes#51676.
Fixes#59294.
Fixes#59678.
Fixes#60007.
Change-Id: Ie51c8e81ade34ec81d69fd7bce1fe0039a470776
Reviewed-on: https://go-review.googlesource.com/c/go/+/495855
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
The test had a 5 second timeout. Running the test on a Darwin system
sometimes took less than 5 seconds but often took up to 8 seconds.
We don't need a timeout anyhow. Instead, use testenv.Command to
run the program, which uses the test timeout.
Fixes#59807
Change-Id: Ibf3eda9702731bf98601782f4abd11c3caa0bf40
Reviewed-on: https://go-review.googlesource.com/c/go/+/494456
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
This reverts CL 485500.
Reason for revert: This breaks internal tests at Google, see b/280861579 and b/280820455.
Change-Id: I426278d400f7611170918fc07c524cb059b9cc55
Reviewed-on: https://go-review.googlesource.com/c/go/+/492995
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Chressie Himpel <chressie@google.com>
It appears to sometimes hang instead of crashing,
which can cause subsequent tests to time out.
For #59947.
Change-Id: Id4ac3c0cd5f7f345334d3e0ed3f48e40b9ff191c
Reviewed-on: https://go-review.googlesource.com/c/go/+/492075
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
This reapplies CL 481061, with the followup fixes in CL 482975, CL 485315, and
CL 485316 incorporated.
CL 481061, by doujiang24 <doujiang24@gmail.com>, speed up C to Go
calls by binding the M to the C thread. See below for its
description.
CL 482975 is a followup fix to a C declaration in testprogcgo.
CL 485315 is a followup fix for x_cgo_getstackbound on Illumos.
CL 485316 is a followup cleanup for ppc64 assembly.
[Original CL 481061 description]
This reapplies CL 392854, with the followup fixes in CL 479255,
CL 479915, and CL 481057 incorporated.
CL 392854, by doujiang24 <doujiang24@gmail.com>, speed up C to Go
calls by binding the M to the C thread. See below for its
description.
CL 479255 is a followup fix for a small bug in ARM assembly code.
CL 479915 is another followup fix to address C to Go calls after
the C code uses some stack, but that CL is also buggy.
CL 481057, by Michael Knyszek, is a followup fix for a memory leak
bug of CL 479915.
[Original CL 392854 description]
In a C thread, it's necessary to acquire an extra M by using needm while invoking a Go function from C. But, needm and dropm are heavy costs due to the signal-related syscalls.
So, we change to not dropm while returning back to C, which means binding the extra M to the C thread until it exits, to avoid needm and dropm on each C to Go call.
Instead, we only dropm while the C thread exits, so the extra M won't leak.
When invoking a Go function from C:
Allocate a pthread variable using pthread_key_create, only once per shared object, and register a thread-exit-time destructor.
And store the g0 of the current m into the thread-specified value of the pthread key, only once per C thread, so that the destructor will put the extra M back onto the extra M list while the C thread exits.
When returning back to C:
Skip dropm in cgocallback, when the pthread variable has been created, so that the extra M will be reused the next time invoke a Go function from C.
This is purely a performance optimization. The old version, in which needm & dropm happen on each cgo call, is still correct too, and we have to keep the old version on systems with cgo but without pthreads, like Windows.
This optimization is significant, and the specific value depends on the OS system and CPU, but in general, it can be considered as 10x faster, for a simple Go function call from a C thread.
For the newly added BenchmarkCGoInCThread, some benchmark results:
1. it's 28x faster, from 3395 ns/op to 121 ns/op, in darwin OS & Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
2. it's 6.5x faster, from 1495 ns/op to 230 ns/op, in Linux OS & Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
[CL 479915 description]
Currently, when C calls into Go the first time, we grab an M
using needm, which sets m.g0's stack bounds using the SP. We don't
know how big the stack is, so we simply assume 32K. Previously,
when the Go function returns to C, we drop the M, and the next
time C calls into Go, we put a new stack bound on the g0 based on
the current SP. After CL 392854, we don't drop the M, and the next
time C calls into Go, we reuse the same g0, without recomputing
the stack bounds. If the C code uses quite a bit of stack space
before calling into Go, the SP may be well below the 32K stack
bound we assumed, so the runtime thinks the g0 stack overflows.
This CL makes needm get a more accurate stack bound from
pthread. (In some platforms this may still be a guess as we don't
know exactly where we are in the C stack), but it is probably
better than simply assuming 32K.
[CL 485500 description]
CL 479915 passed the G to _cgo_getstackbound for direct updates to
gp.stack.lo. A G can be reused on a new thread after the previous thread
exited. This could trigger the C TSAN race detector because it couldn't
see the synchronization in Go (lockextra) preventing the same G from
being used on multiple threads at the same time.
We work around this by passing the address of a stack variable to
_cgo_getstackbound rather than the G. The stack is generally unique per
thread, so TSAN won't see the same address from multiple threads. Even
if stacks are reused across threads by pthread, C TSAN should see the
synchonization in the stack allocator.
A regression test is added to misc/cgo/testsanitizer.
Fixes#51676.
Fixes#59294.
Fixes#59678.
Change-Id: Ic62be31a06ee83568215e875a891df37084e08ca
Reviewed-on: https://go-review.googlesource.com/c/go/+/485500
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TestRaceProf and TestRaceSignal were changed to run on all platforms
that support the race detector as of CL 487575, but the testprogcgo
source files needed to run the test rely on POSIX threads and were
still build-constrained to only linux/amd64 and freebsd/amd64.
Since the C test program appears to require only POSIX APIs, update
the constraint to build the source file on all Unix platforms, and
update the tests to skip on Windows.
This may slightly increase testprogcgo build time on Unix platforms
that do not support the race detector.
Change-Id: I704dd496d475a3cd2e2da2a09c7d2e3bb8e96d02
Reviewed-on: https://go-review.googlesource.com/c/go/+/488115
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Don't try to duplicate the list of targets that support -race.
Change-Id: I889d5c2f4884de89d88f8efdc89608aa73584a8a
Reviewed-on: https://go-review.googlesource.com/c/go/+/487575
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
__tsan_fini will call exit which will call destructors which
may in principle call back into Go functions. Prepare the scheduler
by calling entersyscall before __tsan_fini.
Fixes#59711
Change-Id: Ic4df8fba3014bafa516739408ccfc30aba4f22ad
Reviewed-on: https://go-review.googlesource.com/c/go/+/486615
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
This reverts CL 481061.
Reason for revert: When built with C TSAN, x_cgo_getstackbound triggers
race detection on `g->stacklo` because the synchronization is in Go,
which isn't instrumented.
For #51676.
For #59294.
For #59678.
Change-Id: I38afcda9fcffd6537582a39a5214bc23dc147d47
Reviewed-on: https://go-review.googlesource.com/c/go/+/485275
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
This reapplies CL 392854, with the followup fixes in CL 479255,
CL 479915, and CL 481057 incorporated.
CL 392854, by doujiang24 <doujiang24@gmail.com>, speed up C to Go
calls by binding the M to the C thread. See below for its
description.
CL 479255 is a followup fix for a small bug in ARM assembly code.
CL 479915 is another followup fix to address C to Go calls after
the C code uses some stack, but that CL is also buggy.
CL 481057, by Michael Knyszek, is a followup fix for a memory leak
bug of CL 479915.
[Original CL 392854 description]
In a C thread, it's necessary to acquire an extra M by using needm while invoking a Go function from C. But, needm and dropm are heavy costs due to the signal-related syscalls.
So, we change to not dropm while returning back to C, which means binding the extra M to the C thread until it exits, to avoid needm and dropm on each C to Go call.
Instead, we only dropm while the C thread exits, so the extra M won't leak.
When invoking a Go function from C:
Allocate a pthread variable using pthread_key_create, only once per shared object, and register a thread-exit-time destructor.
And store the g0 of the current m into the thread-specified value of the pthread key, only once per C thread, so that the destructor will put the extra M back onto the extra M list while the C thread exits.
When returning back to C:
Skip dropm in cgocallback, when the pthread variable has been created, so that the extra M will be reused the next time invoke a Go function from C.
This is purely a performance optimization. The old version, in which needm & dropm happen on each cgo call, is still correct too, and we have to keep the old version on systems with cgo but without pthreads, like Windows.
This optimization is significant, and the specific value depends on the OS system and CPU, but in general, it can be considered as 10x faster, for a simple Go function call from a C thread.
For the newly added BenchmarkCGoInCThread, some benchmark results:
1. it's 28x faster, from 3395 ns/op to 121 ns/op, in darwin OS & Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
2. it's 6.5x faster, from 1495 ns/op to 230 ns/op, in Linux OS & Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
[CL 479915 description]
Currently, when C calls into Go the first time, we grab an M
using needm, which sets m.g0's stack bounds using the SP. We don't
know how big the stack is, so we simply assume 32K. Previously,
when the Go function returns to C, we drop the M, and the next
time C calls into Go, we put a new stack bound on the g0 based on
the current SP. After CL 392854, we don't drop the M, and the next
time C calls into Go, we reuse the same g0, without recomputing
the stack bounds. If the C code uses quite a bit of stack space
before calling into Go, the SP may be well below the 32K stack
bound we assumed, so the runtime thinks the g0 stack overflows.
This CL makes needm get a more accurate stack bound from
pthread. (In some platforms this may still be a guess as we don't
know exactly where we are in the C stack), but it is probably
better than simply assuming 32K.
Fixes#51676.
Fixes#59294.
Change-Id: I9bf1400106d5c08ce621d2ed1df3a2d9e3f55494
Reviewed-on: https://go-review.googlesource.com/c/go/+/481061
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Cherry Mui <cherryyz@google.com>
Reviewed-by: DeJiang Zhu (doujiang) <doujiang24@gmail.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
This reverts CL 392854.
Reason for revert: caused #59294, which was derived from google
internal tests. The attempted fix of #59294 caused more breakage.
Change-Id: I5a061561ac2740856b7ecc09725ac28bd30f8bba
Reviewed-on: https://go-review.googlesource.com/c/go/+/481060
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
If a panicking signal (e.g. SIGSEGV) happens on a g0 stack, we're
either in the runtime or running C code. Either way we cannot
recover and sigpanic will immediately throw. Further, injecting a
sigpanic could make the C stack unwinder and the debugger fail to
unwind the stack. So don't inject a sigpanic.
If we have cgo traceback and symbolizer attached, if it panics in
a C function ("CF" for the example below), previously it shows
something like
fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x45f1ef]
runtime stack:
runtime.throw({0x485460?, 0x0?})
.../runtime/panic.go:1076 +0x5c fp=0x7ffd77f60f58 sp=0x7ffd77f60f28 pc=0x42e39c
runtime.sigpanic()
.../runtime/signal_unix.go:821 +0x3e9 fp=0x7ffd77f60fb8 sp=0x7ffd77f60f58 pc=0x442229
goroutine 1 [syscall]:
CF
/tmp/pp/c.c:6 pc=0x45f1ef
runtime.asmcgocall
.../runtime/asm_amd64.s:869 pc=0x458007
runtime.cgocall(0x45f1d0, 0xc000053f70)
.../runtime/cgocall.go:158 +0x51 fp=0xc000053f48 sp=0xc000053f10 pc=0x404551
main._Cfunc_CF()
_cgo_gotypes.go:39 +0x3f fp=0xc000053f70 sp=0xc000053f48 pc=0x45f0bf
Now it shows
SIGSEGV: segmentation violation
PC=0x45f1ef m=0 sigcode=1
signal arrived during cgo execution
goroutine 1 [syscall]:
CF
/tmp/pp/c.c:6 pc=0x45f1ef
runtime.asmcgocall
.../runtime/asm_amd64.s:869 pc=0x458007
runtime.cgocall(0x45f1d0, 0xc00004ef70)
.../runtime/cgocall.go:158 +0x51 fp=0xc00004ef48 sp=0xc00004ef10 pc=0x404551
main._Cfunc_CF()
_cgo_gotypes.go:39 +0x3f fp=0xc00004ef70 sp=0xc00004ef48 pc=0x45f0bf
I think the new one is reasonable.
For #57698.
Change-Id: I4f7af91761374e9b569dce4c7587499d4799137e
Reviewed-on: https://go-review.googlesource.com/c/go/+/462437
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Cherry Mui <cherryyz@google.com>
In a C thread, it's necessary to acquire an extra M by using needm while invoking a Go function from C. But, needm and dropm are heavy costs due to the signal-related syscalls.
So, we change to not dropm while returning back to C, which means binding the extra M to the C thread until it exits, to avoid needm and dropm on each C to Go call.
Instead, we only dropm while the C thread exits, so the extra M won't leak.
When invoking a Go function from C:
Allocate a pthread variable using pthread_key_create, only once per shared object, and register a thread-exit-time destructor.
And store the g0 of the current m into the thread-specified value of the pthread key, only once per C thread, so that the destructor will put the extra M back onto the extra M list while the C thread exits.
When returning back to C:
Skip dropm in cgocallback, when the pthread variable has been created, so that the extra M will be reused the next time invoke a Go function from C.
This is purely a performance optimization. The old version, in which needm & dropm happen on each cgo call, is still correct too, and we have to keep the old version on systems with cgo but without pthreads, like Windows.
This optimization is significant, and the specific value depends on the OS system and CPU, but in general, it can be considered as 10x faster, for a simple Go function call from a C thread.
For the newly added BenchmarkCGoInCThread, some benchmark results:
1. it's 28x faster, from 3395 ns/op to 121 ns/op, in darwin OS & Intel(R) Core(TM) i7-9750H CPU @ 2.60GHz
2. it's 6.5x faster, from 1495 ns/op to 230 ns/op, in Linux OS & Intel(R) Xeon(R) CPU E5-2630 0 @ 2.30GHz
Fixes#51676
Change-Id: I380702fe2f9b6b401b2d6f04b0aba990f4b9ee6c
GitHub-Last-Rev: 93dc64ad98
GitHub-Pull-Request: golang/go#51679
Reviewed-on: https://go-review.googlesource.com/c/go/+/392854
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: thepudds <thepudds1460@gmail.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Many of the tests skipped platforms that build PIE binaries by
default, but (still) lack a central function to report which platforms
those are.
Some of the tests assumed (but did not check for) internal linking
support, or invoked `go tool link` directly without properly
configuring the external linker.
A few of the tests seem to be triggering latent bugs in the linker.
For #58806.
For #58807.
For #58794.
Change-Id: Ie4d06b1597f404590ad2abf978d4c363647407ac
Reviewed-on: https://go-review.googlesource.com/c/go/+/472455
Reviewed-by: Cherry Mui <cherryyz@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Fixes#54778
Change-Id: If9aef0c06b993ef2aedbeea9452297ee9f11fa06
Reviewed-on: https://go-review.googlesource.com/c/go/+/460461
Reviewed-by: Ian Lance Taylor <iant@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Ian Lance Taylor <iant@google.com>
This test was originally Linux-only, but there doesn't seem to be
anything Linux-specific in it.
Change-Id: I0f8519eff5dbed97f5e21e1c8e5ab0d747d51df3
Reviewed-on: https://go-review.googlesource.com/c/go/+/443073
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
This migrates testsigfwd, which uses some one-off build
infrastructure, to be part of the runtime's testprogcgo.
The test is largely unchanged. Because it's part of a larger binary,
this CL renames a few things and gates the constructor-time signal
handler registration on an environment variable. This CL also replaces
an errant fmt.Errorf with fmt.Fprintf.
For #37486, since it eliminates a non-go-test from dist.
Change-Id: I0efd146ea0a0a3f0b361431349a419af0f0ecc61
Reviewed-on: https://go-review.googlesource.com/c/go/+/443068
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Ian Lance Taylor <iant@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
already skips tests in case of the timestamp error, eg. #97757
Change-Id: Ia696e83cba2e3ed50181a8100b964847092a7365
GitHub-Last-Rev: 8e5f607e14
GitHub-Pull-Request: golang/go#55918
Reviewed-on: https://go-review.googlesource.com/c/go/+/435855
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Joel Sing <joel@sing.id.au>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
Extra Ms may lead to the "no consistent ordering of events possible" error when parsing trace file with cgo enabled, since:
1. The gs in the extra Ms may be in `_Gdead` status while starting trace by invoking `runtime.StartTrace`,
2. and these gs will trigger `traceEvGoSysExit` events in `runtime.exitsyscall` when invoking go functions from c,
3. then, the events of those gs are under non-consistent ordering, due to missing the previous events.
Add two events, `traceEvGoCreate` and `traceEvGoInSyscall`, in `runtime.StartTrace`, will make the trace parser happy.
Fixes#29707
Change-Id: I2fd9d1713cda22f0ddb36efe1ab351f88da10881
GitHub-Last-Rev: 7bbfddb81b
GitHub-Pull-Request: golang/go#54974
Reviewed-on: https://go-review.googlesource.com/c/go/+/429858
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
Run-TryBot: xie cui <523516579@qq.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Michael Pratt <mpratt@google.com>
On Linux a signal sent using tgkill will have si_code == SI_TKILL,
not SI_USER. Treat the two cases the same. Add a Linux-specific test.
Change the test to use the C pause function rather than sleeping
for a second, as that achieves the same effect.
This is a roll forward of CL 431255 which was rolled back in CL 431715.
This new version skips flaky tests on more systems, and marks a new method
nosplit.
Change-Id: Ibf2d3e6fc43d63d0a71afa8fcca6a11fda03f291
Reviewed-on: https://go-review.googlesource.com/c/go/+/432136
Auto-Submit: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
On Linux a signal sent using tgkill will have si_code == SI_TKILL,
not SI_USER. Treat the two cases the same. Add a Linux-specific test.
Change the test to use the C pause function rather than sleeping
for a second, as that achieves the same effect.
Change-Id: I2a36646aecabcab9ec42ed9a048b07c2ff0a3987
Reviewed-on: https://go-review.googlesource.com/c/go/+/431255
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Ian Lance Taylor <iant@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Ian Lance Taylor <iant@google.com>
Extra Ms may lead to the "no consistent ordering of events possible" error when parsing trace file with cgo enabled, since:
1. The gs in the extra Ms may be in `_Gdead` status while starting trace by invoking `runtime.StartTrace`,
2. and these gs will trigger `traceEvGoSysExit` events in `runtime.exitsyscall` when invoking go functions from c,
3. then, the events of those gs are under non-consistent ordering, due to missing the previous events.
Add two events, `traceEvGoCreate` and `traceEvGoInSyscall`, in `runtime.StartTrace`, will make the trace parser happy.
Fixes#29707
Change-Id: I7cc4b80822d2c46591304a59c9da2c9fc470f1d0
GitHub-Last-Rev: 445de8eaf3
GitHub-Pull-Request: golang/go#53284
Reviewed-on: https://go-review.googlesource.com/c/go/+/411034
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Auto-Submit: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
CL 390034 changed this throw message to add the goid, breaking the
match.
For #50979.
Change-Id: I52d97695484938701e5b7c269e2caf0c87d44d7a
Reviewed-on: https://go-review.googlesource.com/c/go/+/391139
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
This test has failed on four different builders in the past month.
Moreover, because every Go program depends on "runtime", it is likely
to be run any time a user runs 'go test all' in their own program.
Since the test is known to be flaky, let's skip it to avoid
introducing testing noise until someone has time to investigate. It
seems like we have enough samples in the builder logs to at least
start with.
For #50979
Change-Id: I9748a82fbb97d4ed95d6f474427e5aa6ecdb023d
Reviewed-on: https://go-review.googlesource.com/c/go/+/385154
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Add a regression test for issue 50936 which coerces the runtime into
frequent execution of the cgocall dropg/execute curg assignment race by
making many concurrent cgo calls eligible for P retake by sysmon. This
results in no P during exitsyscall, at which point they will update curg
and will crash if SIGPROF arrives in between updating mp.curg and
mp.curg.m.
This test is conceptually similar to the basic cgo callback test in
aprof.go but with additional concurrency and a sleep in C.
On my machine this test fails ~5% of the time prior to CL 382079.
For #50936.
Change-Id: I21b6c7f2594f9a615a64580ef70a88b692505678
Reviewed-on: https://go-review.googlesource.com/c/go/+/382244
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
This failure mode has been present since at least 2020-06-08. We have
enough information to diagnose it, and further failures don't seem to
be adding any new information at this point: they can only add noise,
both on the Go project's builders and in users' own modules (for
example, when run as part of 'go test all').
For #39457
Change-Id: I2379631da0c8af69598fa61c0cc5ac0ea6ba8267
Reviewed-on: https://go-review.googlesource.com/c/go/+/382395
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Use an enviroment variable rather than a build tag to control starting
a busy loop thread when testprogcgo starts. This lets us skip another
build that invokes the C compiler and linker, which should avoid
timeouts running the runtime tests.
Fixes#44422
Change-Id: I516668d71a373da311d844990236566ff63e6d72
Reviewed-on: https://go-review.googlesource.com/c/go/+/379294
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Also adjust other skips to actually call t.Skip, so that the test
correctly shows as skipped instead of passing.
For #50605
Change-Id: Ied482f231a879224c5a92e2c47a6b21c1593a7da
Reviewed-on: https://go-review.googlesource.com/c/go/+/378554
Trust: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
The VDSO (__kernel_vsyscall) is reachable via
asmcgocall(cgo_start_thread) on linux-386, which causes traceback to
throw.
Fixes#49182.
For #50504.
Change-Id: Idb78cb8de752203ce0ed63c2dbd2d12847338688
Reviewed-on: https://go-review.googlesource.com/c/go/+/376656
Reviewed-by: Cherry Mui <cherryyz@google.com>
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
If the test's main goroutine receives a SIGPROF while creating the
C-owned thread for the test, that sample will appear in the resulting
profile. The root end of that stack will show a set of Go functions. The
leaf end will be the C functions returned by the SetCgoTraceback
handler, which will confuse the test runner.
Add a label to the main goroutine while it calls in to C, so all profile
samples that triggered the SetCgoTraceback handler are either correct,
or can easily be excluded from the test's analysis. (The labels will not
apply to the resulting C-owned thread, which does not use goroutines.)
Fixes#43174
Change-Id: Ica3100ca0f191dcf91b30b0084e8541c5a25689f
Reviewed-on: https://go-review.googlesource.com/c/go/+/370135
Reviewed-by: Michael Pratt <mpratt@google.com>
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
Trust: Michael Knyszek <mknyszek@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
The code was accidentally repeating the TestCgoExternalThreadSIGPROF test.
While we're here remove an obsolete skip on ppc64/linux.
Change-Id: Icdc4032a67aa80fbcfcd7c5c7ab8a6f23f321e2e
Reviewed-on: https://go-review.googlesource.com/c/go/+/366755
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
This test occasionally fails due to a real bug on this platform.
Due to the age of the platform and the rarity of the failure, we do
not believe that the bug is worth working around.
Fixes#43926
Change-Id: Ia227c5afe81fc21b6630813228f976cc3a54013c
Reviewed-on: https://go-review.googlesource.com/c/go/+/366537
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
the CgoPprof tests currently assume that calls to their cgoTraceback
functions are primarily for generating pprof samples and exit early
after receiving two calls.
This is a fragile assumption, as cgoTraceback will be called for _any_
signal received, hence why the test already looks for 2 calls instead of
1.
Still, this has caused flaky failures in two cases:
* #37201, where async preemption signals add additional probability of
receiving non-profiling signals. This was resolved by disabling async
preemption.
* #49401, where some ITIMER_PROF SIGPROF signals are ignored in favor of
per-thread SIGPROF signals.
Rather than attempting to keep plugging holes, this CL drops the fragile
assumption from these tests. Now they simply unconditionally run for the
full 1s before exiting.
Fixes#49401
Change-Id: I16dc9d2f16c2fb511e9db93dd096a402121f86ac
Reviewed-on: https://go-review.googlesource.com/c/go/+/363634
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
Reviewed-by: Rhys Hiltner <rhys@justin.tv>
When acquire a goroutine profile, we stop the world then acquire a
stack trace for each goroutine. When cgo traceback is used, the
traceback code may call the cgo traceback function using cgocall.
As the world is stopped, cgocall will be blocked at exitsyscall,
causing a deadlock. Bypass the scheduler (using asmcgocall) to fix
this.
Change-Id: Ic4e596adc3711310b6a983d73786d697ef15dd72
Reviewed-on: https://go-review.googlesource.com/c/go/+/362757
Trust: Cherry Mui <cherryyz@google.com>
Run-TryBot: Cherry Mui <cherryyz@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
When these packages are released as part of Go 1.18,
Go 1.16 will no longer be supported, so we can remove
the +build tags in these files.
Ran go fix -fix=buildtag std cmd and then reverted the bootstrapDirs
as defined in src/cmd/dist/buildtool.go, which need to continue
to build with Go 1.4 for now.
Also reverted src/vendor and src/cmd/vendor, which will need
to be updated in their own repos first.
Manual changes in runtime/pprof/mprof_test.go to adjust line numbers.
For #41184.
Change-Id: Ic0f93f7091295b6abc76ed5cd6e6746e1280861e
Reviewed-on: https://go-review.googlesource.com/c/go/+/344955
Trust: Russ Cox <rsc@golang.org>
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
This was added in CL 339990.
Change-Id: I4b0f97bf1a3926e37a42f77e149dcab3b7b75a63
Reviewed-on: https://go-review.googlesource.com/c/go/+/359255
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
CL 339990 made this test more strict, exposing pre-existing issues on
these OSes. Skip for now until they can be resolved.
Updates #49182
Change-Id: I3ac400dcd21b801bf4ab4eeb630e23b5c66ba563
Reviewed-on: https://go-review.googlesource.com/c/go/+/359254
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
If we receive an async signal while running in the VDSO, such as a
SIGABRT or SIGSEGV sent from another process, we fail to print the
stacktrace with "runtime: unknown pc <vdso PC>".
We already have machinery to handle SIGPROF in the VDSO, but it isn't
hooked up for other signals. Add it to the general signal traceback
path.
This case is covered by TestSegv by making the test more strict w.r.t.
accepted output.
Fixes#47537
Change-Id: I755585f70e0c23e207e135bc6bd2aa68298e5d24
Reviewed-on: https://go-review.googlesource.com/c/go/+/339990
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
CL 64070 removed lockOSThread from the cgocall path, but didn't update
the signal-in-cgo detection in sighandler. As a result, signals that
arrive during a cgo call are treated like they arrived during Go
execution, breaking the traceback.
Update the cgo detection to fix the backtrace.
Fixes#47522
Change-Id: I61d77ba6465f55e3e6187246d79675ba8467ec23
Reviewed-on: https://go-review.googlesource.com/c/go/+/339989
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
In TestSegv, the t.Run closure captures the loop variable 'test'. Since
the subtest calls t.Parallel, the parent test is allowed to keep
running, changing the loop variable and thus changing the value of
'test' in the subtest.
Change-Id: I021ddc50304de08a341e6ffe486aa54e573d3b94
Reviewed-on: https://go-review.googlesource.com/c/go/+/339911
Trust: Michael Pratt <mpratt@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
When returning from Go to C, it was possible for the goroutine to be
preempted after calling unlockOSThread. This could happen when there
a context function installed by SetCgoTraceback set a non-zero context,
leading to a defer call in cgocallbackg1. The defer function wrapper,
introduced in 1.17 as part of the regabi support, was not nosplit,
and hence was a potential preemption point. If it did get preempted,
the G would move to a new M. It would then attempt to return to C
code on a different stack, typically leading to a SIGSEGV.
Fix this in a simple way by postponing the unlockOSThread until after
the other defer. Also check for the failure condition and fail early,
rather than waiting for a SIGSEGV.
Without the fix to cgocall.go, the test case fails about 50% of the
time on my laptop.
Fixes#47441
Change-Id: Ib8ca13215bd36cddc2a49e86698824a29c6a68ba
Reviewed-on: https://go-review.googlesource.com/c/go/+/338197
Trust: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Michael Pratt
doujiang24
Cherry Mui
Ian Lance Taylor
Chressie Himpel
Bryan C. Mills
Austin Clements
Cuong Manh Le
eric fang
Rhys Hiltner
Russ Cox