The problem was that spans end up in wrong lists after split
(e.g. in h->busy instead of h->central->empty).
Also the span can be non-swept before split,
I don't know what it can cause, but it's safer to operate on swept spans.
Fixes#7544.
R=golang-codereviews, rsc
CC=golang-codereviews, khr
https://golang.org/cl/76160043
Currently processes crash with obscure message.
Say that it's "out of memory".
LGTM=rsc
R=golang-codereviews
CC=golang-codereviews, khr, rsc
https://golang.org/cl/75820045
The Solaris network poller uses event ports, which are
level-triggered. As such, it has to re-arm itself after each
wakeup. The arming mechanism (which runs in its own thread) raced
with the closing of a file descriptor happening in a different
thread. When a network file descriptor is about to be closed,
the network poller is awaken to give it a chance to remove its
association with the file descriptor. Because the poller always
re-armed itself, it raced with code that closed the descriptor.
This change makes the network poller check before re-arming if
the file descriptor is about to be closed, in which case it will
ignore the re-arming request. It uses the per-PollDesc lock in
order to serialize access to the PollDesc.
This change also adds extensive documentation describing the
Solaris implementation of the network poller.
Fixes#7410.
LGTM=dvyukov, iant
R=golang-codereviews, bradfitz, iant, dvyukov, aram.h, gobot
CC=golang-codereviews
https://golang.org/cl/69190044
debug/elf does the same thing, use []byte{} for
any missing sections.
Fixes#7510
LGTM=rsc
R=golang-codereviews, iant
CC=golang-codereviews, rsc
https://golang.org/cl/75230043
Mark free memory blocks as unused.
On amd64 it allows the process to eat all 128 GB of heap
without killing the machine.
LGTM=rsc
R=rsc
CC=golang-codereviews
https://golang.org/cl/74070043
This is especially important for SetPanicOnCrash,
but also useful for e.g. nil deref in mallocgc.
Panics on such crashes can't lead to anything useful,
only to deadlocks, hangs and obscure crashes.
This is a copy of broken but already LGTMed
https://golang.org/cl/68540043/
TBR=rsc
R=rsc
CC=golang-codereviews
https://golang.org/cl/75320043
When we copy stack, we check only new size of the top segment.
This is incorrect, because we can have other segments below it.
LGTM=khr
R=golang-codereviews, khr
CC=golang-codereviews, rsc
https://golang.org/cl/73980045
Calling runtime·cgocall could trigger a GC in the child while
gclock was held by the parent.
Fixes#7511
LGTM=bradfitz, dvyukov, dave
R=golang-codereviews, bradfitz, dvyukov, dave
CC=golang-codereviews, rsc
https://golang.org/cl/75210044
Previously, we wrote "kill" to the process control file
to kill a program. This is problematic because it doesn't
let the program gracefully exit.
This matters especially if the process we're killing is a
Go program. On Unix, sending SIGKILL to a Go program will
automatically kill all runtime threads. On Plan 9, there
are no threads so when the program wants to exit it has to
somehow signal all of the runtime processes. It can't do
this if we mercilessly kill it by writing to it's control
file.
Instead, we now send it a note to invoke it's note handler
and let it perform any cleanup before exiting.
LGTM=rsc
R=rsc, 0intro
CC=golang-codereviews
https://golang.org/cl/74440044
On Plan 9, the kernel disallows the use of floating point
instructions while handling a note. Previously, we worked
around this by using a simple loop in place of memmove.
When I added that work-around, I verified that all paths
from the note handler didn't end up calling memmove. Now
that memclr is using SSE instructions, the same process
will have to be done again.
Instead of doing that, however, this CL just punts and
uses unoptimized functions everywhere on Plan 9.
LGTM=rsc
R=rsc, 0intro
CC=golang-codereviews
https://golang.org/cl/73830044
syscall.naclWrite was missing from sys_nacl_386.s
This gets ./make.bash passing, but doesn't pass validation. I'm not sure if this is the fault of this change, or validation was broken anyway.
LGTM=rsc
R=minux.ma, rsc
CC=golang-codereviews
https://golang.org/cl/74510043
1. Fix the bug that shrinkstack returns memory to heap.
This causes growslice to misbehave (it manually initialized
blocks, and in efence mode shrinkstack's free leads to
partially-initialized blocks coming out of growslice.
Which in turn causes GC to crash while treating the garbage
as Eface/Iface.
2. Enable efence for stack segments.
LGTM=rsc
R=golang-codereviews, rsc
CC=golang-codereviews, khr
https://golang.org/cl/74080043
The garbage collector uses type information to guide the
traversal of the heap. If it sees a field that should be a string,
it marks the object pointed at by the string data pointer as
visited but does not bother to look at the data, because
strings contain bytes, not pointers.
If you save s[len(s):] somewhere, though, the string data pointer
actually points just beyond the string data; if the string data
were exactly the size of an allocated block, the string data
pointer would actually point at the next block. It is incorrect
to mark that next block as visited and not bother to look at
the data, because the next block may be some other type
entirely.
The fix is to ignore strings with zero length during collection:
they are empty and can never become non-empty: the base
pointer will never be used again. The handling of slices already
does this (but using cap instead of len).
This was not a bug in Go 1.2, because until January all string
allocations included a trailing NUL byte not included in the
length, so s[len(s):] still pointed inside the string allocation
(at the NUL).
This bug was causing the crashes in test/run.go. Specifically,
the parsing of a regexp in package regexp/syntax allocated a
[]syntax.Inst with rounded size 1152 bytes. In fact it
allocated many such slices, because during the processing of
test/index2.go it creates thousands of regexps that are all
approximately the same complexity. That takes a long time, and
test/run works on other tests in other goroutines. One such
other test is chan/perm.go, which uses an 1152-byte source
file. test/run reads that file into a []byte and then calls
strings.Split(string(src), "\n"). The string(src) creates an
1152-byte string - and there's a very good chance of it
landing next to one of the many many regexp slices already
allocated - and then because the file ends in a \n,
strings.Split records the tail empty string as the final
element in the slice. A garbage collection happens at this
point, the collection finds that string before encountering
the []syntax.Inst data it now inadvertently points to, and the
[]syntax.Inst data is not scanned for the pointers that it
contains. Each syntax.Inst contains a []rune, those are
missed, and the backing rune arrays are freed for reuse. When
the regexp is later executed, the runes being searched for are
no longer runes at all, and there is no match, even on text
that should match.
On 64-bit machines the pointer in the []rune inside the
syntax.Inst is larger (along with a few other pointers),
pushing the []syntax.Inst backing array into a larger size
class, avoiding the collision with chan/perm.go's
inadvertently sized file.
I expect this was more prevalent on OS X than on Linux or
Windows because those managed to run faster or slower and
didn't overlap index2.go with chan/perm.go as often. On the
ARM systems, we only run one errorcheck test at a time, so
index2 and chan/perm would never overlap.
It is possible that this bug is the root cause of other crashes
as well. For now we only know it is the cause of the test/run crash.
Many thanks to Dmitriy for help debugging.
Fixes#7344.
Fixes#7455.
LGTM=r, dvyukov, dave, iant
R=golang-codereviews, dave, r, dvyukov, delpontej, iant
CC=golang-codereviews, khr
https://golang.org/cl/74250043
For now Note, futexsleep and futexwakeup are designed for threads,
not for processes. The explicit use of UMTX_OP_WAIT_UINT_PRIVATE and
UMTX_OP_WAKE_PRIVATE can avoid unnecessary traversals of VM objects,
to hit undiscovered bugs related to VM system on SMP/SMT/NUMA
environment.
Update #7496
LGTM=iant
R=golang-codereviews, gobot, iant, bradfitz
CC=golang-codereviews
https://golang.org/cl/72760043
Solaris doesn't have struct ip_mreqn, instead it uses struct ip_mreq
and struct group_req with struct sockaddr_storage.
Also fixes incorrect SockaddrDatalink.
Update #7399
LGTM=aram, iant
R=golang-codereviews, aram, gobot, iant
CC=golang-codereviews
https://golang.org/cl/73920043
This CL is a reformulation of CL 73110043 containing only the minimum required to get the nacl builds compiling.
LGTM=bradfitz
R=rsc, bradfitz
CC=golang-codereviews
https://golang.org/cl/74220043
The comment for 'Clean' function is prepended with spaces instead of
a single tab, resulting in visually misaligned comment in the generated
documentation.
LGTM=bradfitz
R=golang-codereviews, bradfitz
CC=golang-codereviews
https://golang.org/cl/73840043
The change to signal_amd64.c from CL 15790043 was not merged correctly.
This CL reapplies the change, renaming the file to signal_amd64x.c and adds the appropriate build tags.
LGTM=iant, bradfitz
R=rsc, iant, bradfitz
CC=golang-codereviews
https://golang.org/cl/72790043
Spans are now private to threads, and the loop
is removed from all other functions.
Remove it from marknogc for consistency.
LGTM=khr, rsc
R=golang-codereviews, bradfitz, khr
CC=golang-codereviews, khr, rsc
https://golang.org/cl/72520043
Not many windows users have perl installed. They can just use
standard go tools instead. Also mkerrors_windows.sh script
removed - we don't add any new "unix" errors to windows
syscall package anymore.
LGTM=rsc
R=golang-codereviews, rsc
CC=golang-codereviews
https://golang.org/cl/41060044
Thanks to Ian for spotting these.
runtime.h: define uintreg correctly.
stack.c: address warning caused by the type of uintreg being 32 bits on amd64p32.
Commentary (mainly for my own use)
nacl/amd64p32 defines a machine with 64bit registers, but address space is limited to a 4gb window (the window is placed randomly inside the full 48 bit virtual address space of a process). To cope with this 6c defines _64BIT and _64BITREG.
_64BITREG is always defined by 6c, so both GOARCH=amd64 and GOARCH=amd64p32 use 64bit wide registers.
However _64BIT itself is only defined when 6c is compiling for amd64 targets. The definition is elided for amd64p32 environments causing int, uint and other arch specific types to revert to their 32bit definitions.
LGTM=iant
R=iant, rsc, remyoudompheng
CC=golang-codereviews
https://golang.org/cl/72860046
gc does not report this as an error, but go/types does.
(I suspect that constructing a closure counts as a reference
to &all in gc's implementation).
This is not a tool bug, since the spec doesn't require
implementations to implement this check, but it does
illustrate that dialect variations are always a nuisance.
LGTM=rsc, bradfitz
R=bradfitz
CC=golang-codereviews, gri, rsc
https://golang.org/cl/73850043
Previously, passing a long duration to ParseDuration could result in
random, even negative, values.
LGTM=r
R=golang-codereviews, bradfitz, r
CC=golang-codereviews
https://golang.org/cl/72120043
CL 69340044 requires that syscall.SO_ERROR be defined on all unix like platforms. Add SO_ERROR to the list of dummy constants in sycall/net_nacl.go.
LGTM=bradfitz
R=iant, rsc, mikioh.mikioh, bradfitz
CC=golang-codereviews
https://golang.org/cl/73100043
If we report a leak, make sure we've waited long enough to be sure.
The new sleep regimen waits 1.05 seconds before failing; the old
one waited 0.005 seconds.
(The single linux/amd64 failure in this test feels more like a
timing problem than a leak. I don't want to spend time on it unless
we're sure.)
LGTM=bradfitz
R=bradfitz
CC=golang-codereviews
https://golang.org/cl/72630043
We provide amd64p32 implementations for md5 and sha1 so we need to exclude amd64p32 from the generic implementations in those packages.
Fixes build once CL 72360044 lands.
LGTM=agl, remyoudompheng
R=rsc, bradfitz, agl, remyoudompheng
CC=golang-codereviews
https://golang.org/cl/72460043
This code being buggy is the only explanation I can come up
with for issue 7325. It's probably not, but the only alternative
is a Windows kernel bug. Comment this out to see what breaks
or gets fixed.
Update #7325
LGTM=bradfitz
R=golang-codereviews, bradfitz
CC=golang-codereviews
https://golang.org/cl/72590044
From the trace it appears that stackalloc is being
called with 0x1800 which is 6k = 4k + (StackSystem=2k).
Make StackSystem 4k too, to make stackalloc happy.
It's already 4k on windows/amd64.
TBR=khr
CC=golang-codereviews
https://golang.org/cl/72600043
There are at least 3 bugs:
1. g->stacksize accounting is broken during copystack/shrinkstack
2. stktop->free is not properly maintained during copystack/shrinkstack
3. stktop->free logic is broken:
we can have stktop->free==FixedStack,
and we will free it into stack cache,
but it actually comes from heap as the result of non-copying segment shrink
This shows as at least spurious races on race builders (maybe something else as well I don't know).
The idea behind the refactoring is to consolidate stacksize and
segment origin logic in stackalloc/stackfree.
Fixes#7490.
LGTM=rsc, khr
R=golang-codereviews, rsc, khr
CC=golang-codereviews
https://golang.org/cl/72440043
Recursive panics leave dangling Panic structs in g->panic stack.
At best it leads to a Defer leak and incorrect output on a subsequent panic.
At worst it arbitrary corrupts heap.
LGTM=rsc
R=rsc
CC=golang-codereviews
https://golang.org/cl/72480043
One reason the sync.Pool finalizer test can fail is that
this function's ef1 contains uninitialized data that just
happens to point at some of the old pool. I've seen this cause
retention of a single pool cache line (32 elements) on arm.
Really we need liveness information for C functions, but
for now we can be more careful about data in long-lived
C functions that block.
LGTM=bradfitz, dvyukov
R=golang-codereviews, bradfitz, dvyukov
CC=golang-codereviews, iant, khr
https://golang.org/cl/72490043
Instead, split the underlying storage in half and
free just half of it.
Shrinking without copying lets us reclaim storage used
by a previously profligate Go routine that has now blocked
inside some C code.
To shrink in place, we need all stacks to be a power of 2 in size.
LGTM=rsc
R=golang-codereviews, rsc
CC=golang-codereviews
https://golang.org/cl/69580044
Two memory allocator bug fixes.
- efence is not maintaining the proper heap metadata
to make eventual memory reuse safe, so use SysFault.
- now that our heap PageSize is 8k but most hardware
uses 4k pages, SysAlloc and SysReserve results must be
explicitly aligned. Do that in a few more call sites and
document this fact in malloc.h.
Fixes#7448.
LGTM=iant
R=golang-codereviews, josharian, iant
CC=dvyukov, golang-codereviews
https://golang.org/cl/71750048
I've just needed the G status on fault to debug runtime bug.
For some reason we print everything except header here.
Make it more informative and consistent.
R=rsc
CC=golang-codereviews
https://golang.org/cl/67870056
Implement custom assembly thunks for hot race calls (memory accesses and function entry/exit).
The thunks extract caller pc, verify that the address is in heap or global and switch to g0 stack.
Before:
ok regexp 3.692s
ok compress/bzip2 9.461s
ok encoding/json 6.380s
After:
ok regexp 2.229s (-40%)
ok compress/bzip2 4.703s (-50%)
ok encoding/json 3.629s (-43%)
For comparison, normal non-race build:
ok regexp 0.348s
ok compress/bzip2 0.304s
ok encoding/json 0.661s
Race build:
ok regexp 2.229s (+540%)
ok compress/bzip2 4.703s (+1447%)
ok encoding/json 3.629s (+449%)
Also removes some race-related special cases from cgocall and scheduler.
In long-term it will allow to remove cyclic runtime/race dependency on cmd/cgo.
Fixes#4249.
Fixes#7460.
Update #6508
Update #6688
R=iant, rsc, bradfitz
CC=golang-codereviews
https://golang.org/cl/55100044
During the glob decoding process interface values are set to concrete
values after a test for assignability. If the assignability test fails
a slightly vague error message is produced. While technically accurate
the error message does not clearly describe the problem.
Rewrite the error message to include the usage of the word assignable,
which makes it clear the concrete value type is not assignable to the
interface value type.
Fixes#6467.
LGTM=r
R=golang-codereviews, rsc, r
CC=golang-codereviews
https://golang.org/cl/71590043
32-bit Windows uses "structured exception handling" (SEH) to
handle hardware faults: that there is a per-thread linked list
of fault handlers maintained in user space instead of
something like Unix's signal handlers. The structures in the
linked list are required to live on the OS stack, and the
usual discipline is that the function that pushes a record
(allocated from the current stack frame) onto the list pops
that record before returning. Not to pop the entry before
returning creates a dangling pointer error: the list head
points to a stack frame that no longer exists.
Go pushes an SEH record in the top frame of every OS thread,
and that record suffices for all Go execution on that thread,
at least until cgo gets involved.
If we call into C using cgo, that called C code may push its
own SEH records, but by the convention it must pop them before
returning back to the Go code. We assume it does, and that's
fine.
If the C code calls back into Go, we want the Go SEH handler
to become active again, not whatever C has set up. So
runtime.callbackasm1, which handles a call from C back into
Go, pushes a new SEH record before calling the Go code and
pops it when the Go code returns. That's also fine.
It can happen that when Go calls C calls Go like this, the
inner Go code panics. We allow a defer in the outer Go to
recover the panic, effectively wiping not only the inner Go
frames but also the C calls. This sequence was not popping the
SEH stack up to what it was before the cgo calls, so it was
creating the dangling pointer warned about above. When
eventually the m stack was used enough to overwrite the
dangling SEH records, the SEH chain was lost, and any future
panic would not end up in Go's handler.
The bug in TestCallbackPanic and friends was thus creating a
situation where TestSetPanicOnFault - which causes a hardware
fault - would not find the Go fault handler and instead crash
the binary.
Add checks to TestCallbackPanicLocked to diagnose the mistake
in that test instead of leaving a bad state for another test
case to stumble over.
Fix bug by restoring SEH chain during deferred "endcgo"
cleanup.
This bug is likely present in Go 1.2.1, but since it depends
on Go calling C calling Go, with the inner Go panicking and
the outer Go recovering the panic, it seems not important
enough to bother fixing before Go 1.3. Certainly no one has
complained.
Fixes#7470.
LGTM=alex.brainman
R=golang-codereviews, alex.brainman
CC=golang-codereviews, iant, khr
https://golang.org/cl/71440043
Regenerate z-files for DragonFly BSD 3.6.
F_DUP_FD_CLOEXEC is now supported, so remove the zero value constant
from types_dragonfly.go so that we use the generated value from the
z-files.
LGTM=mikioh.mikioh
R=golang-codereviews, mikioh.mikioh
CC=golang-codereviews
https://golang.org/cl/70080047
The format of the DragonFly BSD syscalls.master file has changed
slightly - update mksysnum_dragonfly.pl to match.
LGTM=mikioh.mikioh
R=golang-codereviews, mikioh.mikioh
CC=golang-codereviews
https://golang.org/cl/71460044
Disable the "udp" to IPv6 unicast address on the loopback interface
test under DragonFly BSD. This currently returns a local address of
0.0.0.1, rather than an IPv6 address with zone identifier.
Update #7473
LGTM=mikioh.mikioh
R=golang-codereviews, mikioh.mikioh
CC=golang-codereviews
https://golang.org/cl/71500044
Performing multiple connect system calls on a non-blocking socket
under DragonFly BSD does not necessarily result in errors from earlier
connect calls being returned, particularly if we are connecting to
localhost. Instead, once netpoll tells us that the socket is ready,
get the SO_ERROR socket option to see if the connection succeeded
or failed.
Fixes#7474
LGTM=mikioh.mikioh
R=mikioh.mikioh
CC=golang-codereviews
https://golang.org/cl/69340044
Added test cases and expanded test harness to handle token end
positions.
Also: Make sure token end positions are never outside the valid
position range, as was possible in case of parse errors.
Fixes#7458.
LGTM=bradfitz
R=bradfitz
CC=golang-codereviews
https://golang.org/cl/70190046
the crypto/tls revision d3d43f270632 (CL 67010043, requiring ServerName or InsecureSkipVerify) breaks net/smtp,
since it seems impossible to do SMTP via TLS anymore. i've tried to fix this by simply using a tls.Config with
ServerName, instead of a nil *tls.Config. without this fix, doing SMTP with TLS results in error "tls: either
ServerName or InsecureSkipVerify must be specified in the tls.Config".
testing: the new method TestTlsClient(...) sets up a skeletal smtp server with tls capability, and test client
injects a "fake" certificate allowing tls to work on localhost; thus, the modification to SendMail(...) enabling
this.
Fixes#7437.
LGTM=bradfitz
R=golang-codereviews, josharian, bradfitz
CC=golang-codereviews
https://golang.org/cl/70380043
The network connection dies differently from the server's
perspective on (some) Windows when the client goes away. Match
on the common prefix (common to Unix and Windows) instead of
the network error part.
Fixes#7456
LGTM=josharian
R=golang-codereviews, josharian
CC=alex.brainman, golang-codereviews, iant
https://golang.org/cl/70010050
For non-closure functions, the context register is uninitialized
on entry and will not be used, but morestack saves it and then the
garbage collector treats it as live. This can be a source of memory
leaks if the context register points at otherwise dead memory.
Avoid this by introducing a parallel set of morestack functions
that clear the context register, and use those for the non-closure functions.
I hope this will help with some of the finalizer flakiness, but it probably won't.
Fixes#7244.
LGTM=dvyukov
R=khr, dvyukov
CC=golang-codereviews
https://golang.org/cl/71030044
I have one machine where this 25 test run is flaky
and fails ("21 >= 21"), but 50 works everywhere.
LGTM=josharian
R=josharian
CC=golang-codereviews
https://golang.org/cl/67870053
It mentioned true and false for error values. Instead, just
don't mention the error semantics, as they match normal Go
conventions (if error is non-nil, the other value is
meaningless). We generally only document error values when
they're interesting (where non-nil, non-nil is valid, or the
error value can be certain known values or types).
Fixes#7464
LGTM=agl
R=agl
CC=golang-codereviews
https://golang.org/cl/68440044
The flakiness appears to be just in tests, not in the actual code.
Specifically, the many tests call runtime.GC once and expect that
the finalizers will be running in the background when GC returns.
Now that the sweep phase is concurrent with execution, however,
the finalizers will not be run until sweep finishes, which might
be quite a bit later. To force sweep to finish, implement runtime.GC
by calling the actual collection twice. The second will complete the
sweep from the first.
This was reliably broken after a few runs before the CL and now
passes tens of runs:
while GOMAXPROCS=2 ./runtime.test -test.run=Finalizer -test.short \
-test.timeout=300s -test.cpu=$(perl -e 'print ("1,2,4," x 100) . "1"')
do true; done
Fixes#7328.
LGTM=dvyukov
R=dvyukov, dave
CC=golang-codereviews
https://golang.org/cl/71080043
The exception handler runs on the ordinary g stack,
and the stack copier is now trying to do a traceback
across it. That's never been needed before, so it was
unimplemented. Implement it, in all its ugliness.
Fixes windows/amd64 build.
TBR=khr
CC=golang-codereviews
https://golang.org/cl/71030043
1) Move StateHijacked callback earlier, to make it
panic-proof. A Hijack followed by a panic didn't previously
result in ConnState getting fired for StateHijacked. Move it
earlier, to the time of hijack.
2) Don't fire StateActive unless any bytes were read off the
wire while waiting for a request. This means we don't
transition from New or Idle to Active if the client
disconnects or times out. This was documented before, but not
implemented properly.
This CL is required for an pending fix for Issue 7264
LGTM=josharian
R=josharian
CC=golang-codereviews
https://golang.org/cl/69860049
Unfortunately FreeBSD 10 has changed its syscall arguments for
some reasons but as per request at golang-dev this CL does not
generate some structures, syscall numbers and constants as
compatible to FreeBSD 10 as follows:
Structure: Stat_t, IfData, IfMsghdr are based on 8-STABLE
Syscall number: Capsicum API is based on 9-STABLE
Constant: IFT_CARP, SIOCAIFADDR, SIOCSIFPHYADDR are based on 9-STABLE
Update #7193
FreeBSD 10 breaking changes:
r205792: Rename st_*timespec fields to st_*tim for POSIX 2008
compliance.
http://svnweb.freebsd.org/base?view=revision&revision=205792
r254804: Restructure the mbuf pkthdr to make it fit for upcoming
capabilities and features.
http://svnweb.freebsd.org/base?view=revision&revision=254804
r255219: Change the cap_rights_t type from uint64_t to a structure
that we can extend in the future in a backward compatible (API and
ABI) way.
http://svnweb.freebsd.org/base?view=revision&revision=255219
LGTM=iant
R=golang-codereviews, rsc, minux.ma, gobot, iant
CC=golang-codereviews
https://golang.org/cl/56770044