Convert the part that uses relocations to use loader.ExtReloc
directly. It still uses sym.Symbols for now, but not sym.Relocs.
This reduces some memory usage: linking cmd/compile with external
linking,
name old allocs/op new allocs/op delta
Loadlibfull_GC 52.2MB ± 0% 13.9MB ± 0% -73.40% (p=0.008 n=5+5)
name old live-B new live-B delta
Loadlibfull_GC 75.5M ± 0% 61.9M ± 0% -18.02% (p=0.008 n=5+5)
Change-Id: I317ecbf516063c42b255b2caba310ea6281342d7
Reviewed-on: https://go-review.googlesource.com/c/go/+/231319
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Support external linking for the new reloc pass as well, and
enable it on AMD64 and 386.
Change-Id: Ia71aec3d7c14e9d661e0748d2e988f29f220d1e1
Reviewed-on: https://go-review.googlesource.com/c/go/+/230308
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Only enabled for AMD64 when internal linking for now.
Change-Id: I2aa9ee47c0f7413ea7bbcdd31b8317c14220bba3
Reviewed-on: https://go-review.googlesource.com/c/go/+/230302
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Make Wasm more like other architectures, writing data sections to
heap in Asmb instead of Asmb2. Then we can remove the
copy-on-write logic in applying relocations.
Change-Id: I26d5315ea9fba032fe4bdb9b5c7fe483611c4373
Reviewed-on: https://go-review.googlesource.com/c/go/+/230465
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
This change is just the mechanical work of moving the wavefront past
address.
Change-Id: I519ec49fa8ba50760c7d23fc084fcd3bb0544546
Reviewed-on: https://go-review.googlesource.com/c/go/+/229700
Run-TryBot: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This reverts CL 229994.
Reason for revert: break AIX build.
This is nice to have but isn't critical. We can revisit later.
Change-Id: Ifc56a0a4c0fb36859cf7666ab149e25e0e5d4cc0
Reviewed-on: https://go-review.googlesource.com/c/go/+/230459
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
symGroupType was needed for dodata. Now that we have converted
dodata to using the loader, stop overwriting it.
Change-Id: Ie94109c0b35dd2f71a19ebb38f8cf20b6a37c624
Reviewed-on: https://go-review.googlesource.com/c/go/+/229994
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Add linux/{ppc64,ppc64le} and aix/ppc64 arch support for the new
dodata() phase.
This completes the picture in terms of architecture support for the
new dodata(), but to be safe this patch leaves the command line flag
in place there are problems on the builders (especially given that we
have a dead aix-ppc64 builder).
Change-Id: I78da615c3b540d8925ed7b3226e199280eb7451d
Reviewed-on: https://go-review.googlesource.com/c/go/+/229983
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Currently, we run Asmb before reloc, except on Wasm, where the
order is reversed. However, Asmb is no-op on Wasm. So we can
always run Asmb first.
Change-Id: Ifb8989d8150ebdd5777deb05cbccec16f8e36d82
Reviewed-on: https://go-review.googlesource.com/c/go/+/229990
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
They also don't need to do anything for Adddynrel. So we can just
enable it.
Change-Id: If85fceca63a7b3cb5a09e5db224c3018060e86de
Reviewed-on: https://go-review.googlesource.com/c/go/+/229993
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Recreation of CL 229863 that was removed from the repo because it
included the linker binary.
Change-Id: I5e96afa079b1217df6e7cba63a107546bd96ef76
Reviewed-on: https://go-review.googlesource.com/c/go/+/230028
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Add elf/ARM arch support for the new dodata() phase.
Change-Id: Iadd772b01036c6c5be95bcc6017f6c05d45a24c0
Reviewed-on: https://go-review.googlesource.com/c/go/+/229868
Reviewed-by: Cherry Zhang <cherryyz@google.com>
They don't have fancy Adddynrel stuff, so we can just enable it.
Change-Id: I84082c3187d8a9ffa3a9c5458959794df0e3c2b6
Reviewed-on: https://go-review.googlesource.com/c/go/+/230030
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
This probably breaks darwin/arm64. Will fix.
Change-Id: I8be168985124f971e9d8ab5bc95c303336dd705b
Reviewed-on: https://go-review.googlesource.com/c/go/+/230019
Reviewed-by: Than McIntosh <thanm@google.com>
Previous CL introduced index fingerprint in the object files.
This CL implements the second part: checking fingerprint
consistency in the linker when packages are loaded.
Change-Id: I05dd4c4045a65adfd95e77b625d6c75a7a70e4f1
Reviewed-on: https://go-review.googlesource.com/c/go/+/229618
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
So we can use it to set per-OS flags.
Also set flagnewDoData after archinit, where IsELF is set.
This should correct the logic of setting flagnewDoData.
Change-Id: I18c7252f141aa35119005c252becc9d7cb74f2f7
Reviewed-on: https://go-review.googlesource.com/c/go/+/229867
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Add elf/386 arch support for the new dodata() phase.
Change-Id: I78341dfe70a90719d95c0044183980f348a3369f
Reviewed-on: https://go-review.googlesource.com/c/go/+/229797
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
It doesn't do what it says. It has been like that since Go 1.4.
The current ouput is pretty useless. Remove it.
Change-Id: Id9b4ba04139aaf7ea59acbd51428b1c992115389
Reviewed-on: https://go-review.googlesource.com/c/go/+/229859
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
This patch begins the work of converting the linker's dodata phase to
work with loader APIs. Passes all.bash on linux/amd64, but hasn't been
tested on anything else (more arch-specific code needs to be written).
Use of the new dodata() phase is currently gated by a temporary
command line flag ("-newdodata"), and there is code in the linker's
main routine to insure that we only use the new version for the right
GOOS/GOARCH (currently restricted to ELF + AMD64).
Change-Id: Ied3966677d2a450bc3e0990e0f519b3fceaab806
Reviewed-on: https://go-review.googlesource.com/c/go/+/229706
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
It is no longer needed as we have converted the fieldtrack pass
to using the loader.
Also free loader.Reachparent after we are done with it.
Change-Id: Ibc4b29f282e1e4aea363a1b549755e31f84b0295
Reviewed-on: https://go-review.googlesource.com/c/go/+/229322
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
This is more or less a direct translation, to get things going.
There are more things we can do to make it better, especially on
the handling of container symbols.
Change-Id: I11a0087e402be8d42b9d06869385ead531755272
Reviewed-on: https://go-review.googlesource.com/c/go/+/229125
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Recreation of CL 228317.
The problem with that original CL was a late requested change,
reordering reloc and asmb, resulting in symbols having stale pointers to
their data. I've fixed this by preallocating the heap variable in OutBuf
for platforms w/o mmap.
Change-Id: Icdb392ac2c8d6518830f4c84cf422e78b8ab68c7
Reviewed-on: https://go-review.googlesource.com/c/go/+/228782
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Hoist dwarfGenerateDebugSyms call up out of dodata to before
loadlibfull. This required a couple of small tweaks to the
loader and to loadlibfull.
Change-Id: I48ffb450d2e48b9e55775b73a6debcd27dbb7b9c
Reviewed-on: https://go-review.googlesource.com/c/go/+/228221
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Convert the linker's findfunctab phase to use the new loader APIs.
Change-Id: Ia980a85963fe2e7c554c212c0cc89208272264bd
Reviewed-on: https://go-review.googlesource.com/c/go/+/227842
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Rework the linker's pcln phase to work with the new loader. As part of
this set of changes the handling of "go.file..." symbols has been
revised somewhat -- previously they were treated as always live in the
loader, and now we no longer do this.
The original plan had been to have the new implementation generate
nameless "inltree" symbols, however the plan now is to keep them
named for now and convert them to nameless in a subsequent patch.
Change-Id: If71c93ff1f146dbb63b6ee2546308acdc94b643c
Reviewed-on: https://go-review.googlesource.com/c/go/+/227759
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Implement text address assignment and trampoline generation using
the loader.
Note: the trampoline insertion part doesn't actually work. It
also needs to propagate Aux symbols for external symbols in
LoadFull. But it won't be needed after converting pclntab
generation, so I'll leave it out for now. This could break
linking large binaries on PPC64 and ARM.
Change-Id: Ie46a35b25d7c027983dd877207cfa8f67c32530b
Reviewed-on: https://go-review.googlesource.com/c/go/+/227482
Reviewed-by: Than McIntosh <thanm@google.com>
Convert the ppc64 architecture's version of gentext to use the new
loader APIs.
Change-Id: Ib4af2608f4b246cb6dde07ceaa4a1f7ced45a700
Reviewed-on: https://go-review.googlesource.com/c/go/+/227021
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Begin the job of converting the linker's "gentext" phase over to use
loader APIs. This patch includes most architectures except for s390x
and PPC (these will be added in subsequent patches, since they require
a couple of loader changes first).
Change-Id: Ic7f55c207dcdbbba657330ef007a72ff7c837416
Reviewed-on: https://go-review.googlesource.com/c/go/+/227017
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reorganize the linker phase ordering so that addexport() runs before
loadlibfull. In previous CLs addexport() was changed to use loader
APIs but then copy back its work into sym.Symbol, so this change
removes the copying/shim code in question.
Change-Id: I17314a90007909e6242ee00e26393f3e4a02cf25
Reviewed-on: https://go-review.googlesource.com/c/go/+/226362
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Add some new loader.Sym equivalents to the archSyms struct so that we
can run setArchSyms earlier in the pipeline, and add a "mode" to
setArchSyms to control whether it should create loader.Sym symbols or
their *sym.Symbol equivalents.
These change needed for a subsequent patch in which addexport() is run
earlier as well
Change-Id: I0475c9388c39f13e045dd4aa9c90eaec42624810
Reviewed-on: https://go-review.googlesource.com/c/go/+/226361
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Rewrite setupdynexp to work with loader.Sym, and move the call to it
before the body of loadlibfull. After loadlibfull is complete,
construct the old *sym.Symbol version of dynexp, since not all all
clients that access this list are converted to the loader APIs.
Change-Id: I347d24958e2f3e2332fbe33f2eb6ec25cc126bdb
Reviewed-on: https://go-review.googlesource.com/c/go/+/224382
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Use symbol's Extname, instead of symbol renaming, for the mangled
names.
The old symbol Rename has an interesting logic of "merging"
symbols, when a symbol is renamed to the name of an existing
symbol. It turns out that this is needed for linking against
shared libraries, where the Go object has a reference to a symbol
with the original name, and the shared libary provides a symbol
under the mangled name. Implement this logic with the loader.
Change-Id: Ib95d7a9c93a52f8e02f4a51ac67240d6ebfc1c6a
Reviewed-on: https://go-review.googlesource.com/c/go/+/224939
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
Now we can choose the old object file format by setting
-gcflags=all=-go115newobj=false -asmflags=all=-go115newobj=false -ldflags=all=-go115newobj=false
Tested that setting all three to default false and it still works.
Change-Id: I9514b62a676916cc383b8afa389489fe7b8fa2bf
Reviewed-on: https://go-review.googlesource.com/c/go/+/224625
Reviewed-by: Than McIntosh <thanm@google.com>
Remove the temporary "-newdw2" linker command line option, along with
the remainder of the legacy sym.Symbol based DWARF generation code.
Change-Id: I86c0581dd021cd4e2209ca9bc45f34037d42323c
Reviewed-on: https://go-review.googlesource.com/c/go/+/222766
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
dodata is a pile of dependencies on *Link. Pull some of these
dependencies out, and tighten up the visibility on the functions.
Change-Id: Id213f19ae7b63dd246ddb47b77ffde99f615aa80
Reviewed-on: https://go-review.googlesource.com/c/go/+/222061
Reviewed-by: Than McIntosh <thanm@google.com>
This converts the second portion of DWARF-gen (dwarfGenerateDebugSyms
function and friends) to use the loader, as opposed to sym.Symbols.
Second phase is invoked as part of dodata(), which is fairly far along
in the linker, meaning that the "converted to loader.Symbol" wavefront
has not yet reached that point.
To account for this, the patch contains some shim code that runs
initially to copy back things from sym.Symbols into the loader
(notable symbol values), and then a second hook to be code after
dwarf-gen is complete to that copies the results back into the
appropriate sym.Symbols. This code is obviously temporary, but needed
for the time being.
Change-Id: I9b830b08b16480a1a5230cb52b592db1c550af18
Reviewed-on: https://go-review.googlesource.com/c/go/+/220987
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Also correct a few places I hadn't switched over to Target from *Link
context.
Change-Id: I6de6b39723762de3fa5f8d12800085421aca76fb
Reviewed-on: https://go-review.googlesource.com/c/go/+/220840
Reviewed-by: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Remove the "-newdw" command line option, a temporary placeholder/gate
for the first phase of new DWARF gen, along with some portions of the
sym.Symbol based DWARF type gen code. The remainder of the old code
will be removed in a subsequent patch.
Change-Id: I7870539f020fbf05d0345d3af2acf383fae94892
Reviewed-on: https://go-review.googlesource.com/c/go/+/220984
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Converts the portion of DWARF generation that deals with creation of
type DIEs and constant DIEs to use the new loader interfaces. Creation
of subprogram DIE and compilation unit DIE content still operates on
sym.Symbols at the moment, and happens much later in the linker.
The new code for type DIE generation is gated/guarded by the linker
flag "-newdw", which currently defaults to true. At some point in the
near future this flag should be removed, but it is handy for triage at
the moment.
This patch also includes shim code designed to run after loadlibfull()
that walks through the DIE chains and to converts loader.Sym
references back into sym.Symbol references for the remainder of the
compilation, since the second phase of DWARF has not yet been
converted.
Change-Id: I681a00fb8a1f3c37884a79b373d86411332e07c8
Reviewed-on: https://go-review.googlesource.com/c/go/+/208230
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
No real reason to keep the old code around as the new stuff is working
just fine.
Change-Id: I5e886f8274344738663a0ead181c5d58d9f5a45f
Reviewed-on: https://go-review.googlesource.com/c/go/+/215997
Run-TryBot: Jeremy Faller <jeremy@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Flag flip to make "-newldelf" linker option default to true. Next step
will be to remove the old code.
Change-Id: Ie20637ae2f459b32a9aa30e711f51549da28ffd0
Reviewed-on: https://go-review.googlesource.com/c/go/+/213427
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Switch the linker's Link.linksetup method to work with loader.Sym
instead of sym.Symbols. Currently enabled when the new ELF host object
loader is turned on.
Change-Id: I336cc9f36166767baac574455531e195b6f1ac57
Reviewed-on: https://go-review.googlesource.com/c/go/+/213423
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
This is a rewrite of the ELF host object loader to use just the Loader
interfaces for symbol creation, without constructing sym.Symbols. At
the moment this is gated under the temporary linker command line
option "-newldelf". This version is able to get through all.bash
on linux/amd64.
Change-Id: I99f41368f75b0df9e35ef3c2cf2a702b732540c6
Reviewed-on: https://go-review.googlesource.com/c/go/+/210779
Run-TryBot: Than McIntosh <thanm@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
There are more cleanups to do, but I want to keep this CL mostly
a pure deletion.
Change-Id: I30f4891a2ea54545fd6b83041746ab65895537e1
Reviewed-on: https://go-review.googlesource.com/c/go/+/206558
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
In the dev.link branch we implemented the new object file format
and (part of) the linker improvements described in
https://golang.org/s/better-linker
The new object file is index-based and provides random access.
The linker maps the object files into read-only memory, and
access symbols on-demand using indices, as opposed to reading
all object files sequentially into the heap with the old format.
The linker carries symbol informations using indices (as opposed
to Symbol data structure). Symbols are created after the
reachability analysis, and only created for reachable symbols.
This reduces the linker's memory usage.
Linking cmd/compile, it creates ~25% fewer Symbols, and reduces
memory usage (inuse_space) by ~15%. (More results from Than.)
Currently, both the old and new object file formats are supported.
The old format is used by default. The new format can be turned
on by using the compiler/assembler/linker's -newobj flag. Note
that the flag needs to be specified consistently to all
compilations, i.e.
go build -gcflags=all=-newobj -asmflags=all=-newobj -ldflags=-newobj
Change-Id: Ia0e35306b5b9b5b19fdc7fa7c602d4ce36fa6abd
Flip back to the old object files for Go 1.14.
Change-Id: I4ad499460fb7156b63fc63e9c6ea4f7099e20af2
Reviewed-on: https://go-review.googlesource.com/c/go/+/204098
Run-TryBot: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
When linking a Go archive, if the archiver invocation is the very last
thing that needs to happen in the link (no "atexit" cleanups required
remove the locally created tmpdir) then call syscall.Exec to invoke
the archiver command instead of the usual exec.Command. This has the
effect of reducing peak memory use for the linker overall, since we
don't be holding onto all of the linker's live memory while the
archiver is running.
Change-Id: Ibbe22d8d67a70cc2a4f91c68aab56d19fb77c393
Reviewed-on: https://go-review.googlesource.com/c/go/+/203821
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Also a similar 'elapsed' function and its usages were deleted.
Fixes#19865.
Change-Id: Ib125365e69cf2eda60de64fa74290c8c7d1fd65a
Reviewed-on: https://go-review.googlesource.com/c/go/+/171730
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
Switch the default to new object files.
Internal linking cgo is disabled for now, as it does not work yet
in newobj mode.
Shared libraries are also broken.
Disable some tests that are known broken for now.
Change-Id: I8ca74793423861d607a2aa7b0d89a4f4d4ca7671
Reviewed-on: https://go-review.googlesource.com/c/go/+/200161
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
If the new object file format is used, now we load full symbol
contents after the deadcode pass, for reachable symbols only.
We still load some informations early, like relocations and the
contents of type symbols, which are used in the deadcode pass.
If we rewrite deadcode to use index directly, we could delay more
of the loading (to sym.Symbol), and perhaps delay the creation of
sym.Symbol.
TODO: internal linking with host objects doesn't work yet.
Change-Id: I7d4880e8f150e8709ffac277e62191623440e4cf
Reviewed-on: https://go-review.googlesource.com/c/go/+/197258
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
This is in preparation of delaying loading symbol content to
after dead code elimination.
Change-Id: Ia4ce82d9a42cc1a76c82f6d0eb9a54b0325ee3ba
Reviewed-on: https://go-review.googlesource.com/c/go/+/197061
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jeremy Faller <jeremy@golang.org>
Parse new object file format in the linker. At least we can link
a hello-world program.
Add a basic "loader", which handles symbol references in the
object file.
- mapping between local and global indices
- resolve by-name references
(TODO: the overwrite logic isn't implemented yet)
Currently we still create sym.Symbol rather early, and, after all
the object files are loaded and indexed references are resolved,
add all symbols to sym.Symbols.
The code here is probably not going in the final version. This
is basically only for debugging purposes -- to make sure the
writer and the reader work as expected.
Change-Id: I895aeea68326fabdb7e5aa1371b8cac7211a09dd
Reviewed-on: https://go-review.googlesource.com/c/go/+/196032
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Than McIntosh <thanm@google.com>
compilebench depends on the legacy heap profile format to read the
allocation stats of build tools. We're adding a benchmark for the
linker to compilebench, so we need the linker to emit the heap profile
in the legacy format.
This is the linker equivalent of CL 35484, which did this for the
compiler.
Change-Id: I16ad60c4f0fd80b4b6d608a5677ebe04e1fb5e5a
Reviewed-on: https://go-review.googlesource.com/c/go/+/176057
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
It is useful to be able to dig the Go version out of a binary,
even a stripped binary. rsc.io/goversion does this for x86
binaries by disassembling the binary to find runtime/proc.go's
startup-time reference to runtime.buildVersion's address.
That approach is quite fragile: the implementation doesn't
work for non-x86 and must be updated as the generated
code changes.
rsc.io/goversion finds the module version string by looking
for random 16-byte framing around the actual string.
This is less fragile but fairly kludgy and requires scanning
the entire data segment.
cmd/buildid finds the build ID by looking for an ELF note
or else falling back to scanning the beginning of the text
segment for a magic string. This has proved quite reliable
and doesn't require scanning much of the binary.
This CL makes it possible to find the Go and module versions
using a scan more like the build ID scan: a symbol early in
the writable data segment starts with a magic number and
then has pointers to the two string variables.
Setting up for "go version <binary>".
For #31624.
Change-Id: I78ea8c52fe1686b5cc5a829ca5f198104d10ebf0
Reviewed-on: https://go-review.googlesource.com/c/go/+/173342
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Jay Conrod <jayconrod@google.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Move the phase of applying relocations later, after the sections
and segments are written to the mmap'd output region. Then apply
relocations directly in the output region, instead of the input.
So the input slices we read in don't need to be modified.
This is in preparation for mmap'ing input files read-only.
Change-Id: If9c80657b4469da36aec5a9ab6acf664f5af8fa0
Reviewed-on: https://go-review.googlesource.com/c/go/+/170739
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
We are preparing for applying relocations to the output buffer.
However, for DWARF compression, relocations need to be applied
before compression, but we don't have an output buffer at that
time. We also cannot delay DWARF compression to when we mmap the
output file, because we need the size of the DWARF sections to
compute the file size.
Instead of applying all the relocations together, we apply
relocations in DWARF sections one symbol at a time, right before
it is writing out for compression. As the symbol content may be
in read-only memory (in the future), we use a temporary buffer
for applying the relocations, and immediately write it out.
If compression is not used, relocations are still applied all
together.
This is in preparation for mmap'ing input files read-only.
Change-Id: Iae6d2dd71313897d5054bcc458d3bb78075b30c3
Reviewed-on: https://go-review.googlesource.com/c/go/+/171397
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Use mmap for writing most of the output file content,
specifically, the sections and segments. After layout, we
already know the sizes and file offsets for the sections and
segments. So we can just write the bytes by copying to a mmap'd
backing store.
The writing of the output file is split into two parts. The first
part writes the sections and segments to the mmap'd region. The
second part writes some extra content, for which we don't know
the size, so we use direct file IO.
This is in preparation for mmap'ing input files read-only.
Change-Id: I9f3b4616a9f96bfd5c940d74c50aacd6d330f7d2
Reviewed-on: https://go-review.googlesource.com/c/go/+/170738
Run-TryBot: Cherry Zhang <cherryyz@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Update the recently-added '-strictdups' sanity checking to avoid
failing the link in cases where we have objects feeding into the link
with a mix of command line flags (and in particular some with "-N" and
some without). This scenario will trigger errors/warnings due to
inlinable functions and wrapper functions that have different sizes
due to presence or lack of optimization.
Update #31034.
Change-Id: I1dd9e37c2f9bea5da0ab82e32e6fc210aebf6a65
Reviewed-on: https://go-review.googlesource.com/c/go/+/169160
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: David Chase <drchase@google.com>
Introduce a new linker command line option "-strictdups", which
enables sanity checking of "ok to duplicate" symbols, especially
DWARF info symbols. Acceptable values are 0 (no checking) 1 (issue
warnings) and 2 (issue a fatal error checks fail).
Currently if we read a DWARF symbol (such as "go.info.PKG.FUNCTION")
from one object file, and then encounter the same symbol later on
while reading another object file, we simply discard the second one
and move on with the link, since the two should in theory be
identical.
If as a result of a compiler bug we wind up with symbols that are not
identical, this tends to (silently) result in incorrect DWARF
generation, which may or may not be discovered depending on who is
consuming the DWARF and what's being done with it.
When this option is turned on, at the point where a duplicate
symbol is detected in the object file reader, we check to make sure
that the length/contents of the symbol are the same as the previously
read symbol, and print a descriptive warning (or error) if not.
For the time being this can be used for one-off testing to find
problems; at some point it would be nice if we can enable it by
default.
Updates #30908.
Change-Id: I64c4e07c326b4572db674ff17c93307e2eec607c
Reviewed-on: https://go-review.googlesource.com/c/go/+/168410
Run-TryBot: Than McIntosh <thanm@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Keith Randall <khr@golang.org>
FlagDataAddr is a vestige from git commit 0cafb9e (2008;
no Gerrit CL number).
It was never used but unfortunately setting it would
cause a spurious warning:
warning: -D<value> is ignored because of -R0x1000
yet if -R was unset e.g. -R=0, the linker would crash
with a divide by zero runtime panic.
Fixes#28921
Change-Id: Ia910399bc269337a9a860f3a26cd48fae6e62724
Reviewed-on: https://go-review.googlesource.com/c/151021
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
This commit adds AIX operating system to cmd/link package for ppc64
architecture.
Updates: #25893
Change-Id: I349e0a2658c31919b72117b62c4c9935c9af07c0
Reviewed-on: https://go-review.googlesource.com/c/138730
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
.rel symbol type is sym.SELFROSECT, and that makes .rel written
into .rdata section. But .rel stores code - jump table used for
external C functions. So we have to mark whole .rdata section
as executable (IMAGE_SCN_MEM_EXECUTE), because of .rel presence
in it.
Move .rel into .text section, and make .rdata section non executable.
I also had to move code that adjusted the size of .rel symbol
before calling textaddress, otherwise textaddress would not
calculate size of .text section correctly.
Fixes#25926
Change-Id: I4962f5de7b367410154c8709adfcd8472de9ac1a
Reviewed-on: https://go-review.googlesource.com/c/125455
Run-TryBot: Alex Brainman <alex.brainman@gmail.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Splits part of dwarfgeneratedebugsyms into a new function,
dwarfGenerateDebugInfo which is called between deadcode elimination
and type name mangling.
This function takes care of collecting and processing the DIEs for
all functions and package-level variables and also generates DIEs
for all types used in the program.
Fixes#23733
Change-Id: I75ef0608fbed2dffc3be7a477f1b03e7e740ec61
Reviewed-on: https://go-review.googlesource.com/111237
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Moves type name mangling after deadcode elimination. The motivation for
doing this is to create a space between deadcode elimination and type name
mangling where DWARF generation for types and variables can exist, to fix
issue #23733.
Change-Id: I9db8ecc0f4efe3df6c1e4025f02642fd452f9a39
Reviewed-on: https://go-review.googlesource.com/111236
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Cherry Zhang <cherryyz@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
The sym.Symbol 'Reachparent' field is used only when field tracking
is enabled. So as to use less memory for the common case where
field tracking is not enabled, remove this field and use a side
table stored in the context to achieve the same functionality.
Updates #26186
Change-Id: Idc5f8b0aa323689d4d51dddb5d1b0341a37bb7d2
Reviewed-on: https://go-review.googlesource.com/121915
Reviewed-by: Ian Lance Taylor <iant@golang.org>
We want to compress DWARF even on macOS, but the native toolchain isn't
going to understand it. Add a flag that can be used to disable
compression, then add Darwin to the whitelist used during internal
linking.
Unlike GNU ld, the Darwin linker doesn't have a handy linker flag to do
compression. But since we're already doing surgery to put the DWARF in
the output executable in the first place, compressing it at the same
time isn't unduly difficult. This does have the slightly odd effect of
compressing some Apple proprietary debug sections, which absolutely
nothing will understand. Leaving them uncompressed didn't make much
sense, though, since I doubt they're useful without (say) __debug_info.
Updates #11799
Change-Id: Ie00b0215c630a798c59d009a641e2d13f0e7ea01
Reviewed-on: https://go-review.googlesource.com/120155
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Forked from CL 111895.
The trickiest part of this is that the binary layout code (blk,
elfshbits, and various other things) assumes a constant offset between
symbols' and sections' file locations and their virtual addresses.
Compression, of course, breaks this constant offset. But we need to
assign virtual addresses to everything before compression in order to
resolve relocations before compression. As a result, compression needs
to re-compute the "address" of the DWARF sections and symbols based on
their compressed size. Luckily, these are at the end of the file, so
this doesn't perturb any other sections or symbols. (And there is, of
course, a surprising amount of code that assumes the DWARF segment
comes last, so what's one more place?)
Relevant benchmarks:
name old time/op new time/op delta
StdCmd 10.3s ± 2% 10.8s ± 1% +5.43% (p=0.000 n=30+30)
name old text-bytes new text-bytes delta
HelloSize 746kB ± 0% 746kB ± 0% ~ (all equal)
CmdGoSize 8.41MB ± 0% 8.41MB ± 0% ~ (all equal)
[Geo mean] 2.50MB 2.50MB +0.00%
name old data-bytes new data-bytes delta
HelloSize 10.6kB ± 0% 10.6kB ± 0% ~ (all equal)
CmdGoSize 252kB ± 0% 252kB ± 0% ~ (all equal)
[Geo mean] 51.5kB 51.5kB +0.00%
name old bss-bytes new bss-bytes delta
HelloSize 125kB ± 0% 125kB ± 0% ~ (all equal)
CmdGoSize 145kB ± 0% 145kB ± 0% ~ (all equal)
[Geo mean] 135kB 135kB +0.00%
name old exe-bytes new exe-bytes delta
HelloSize 1.60MB ± 0% 1.05MB ± 0% -34.39% (p=0.000 n=30+30)
CmdGoSize 16.5MB ± 0% 11.3MB ± 0% -31.76% (p=0.000 n=30+30)
[Geo mean] 5.14MB 3.44MB -33.08%
Fixes#11799.
Updates #6853.
Change-Id: I64197afe4c01a237523a943088051ee056331c6f
Reviewed-on: https://go-review.googlesource.com/118276
Run-TryBot: Heschi Kreinick <heschi@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Currently these two forms of layout are done in a single pass. This
makes it difficult to compress DWARF sections because that must be
done after relocations are applied, which must happen after virtual
address layout, but we can't layout the file until we've compressed
the DWARF sections.
Fix this by separating the two layout steps. In the process, we can
also unify the copy-pasted code in Link.address to compute file
offsets, which currently has some unnecessary variation.
Unlike the current file offset computation, which depends on virtual
addresses, the new computation only uses file offsets and sizes. This
will let us compress the file representation of a segment and create
the file layout based on its on-disk size rather than its original
in-memory size.
Tested by comparing the test binary for the "strings" package on all
supported GOOS/GOARCH combinations. All binaries are identical
(except, of course, their build IDs).
This is a second attempt at CL 111682.
For #11799.
Fixes#25863.
Change-Id: If09f28771bb4d78dd392fd58b8d7c9d5f22b0b9e
Reviewed-on: https://go-review.googlesource.com/118716
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Currently these two forms of layout are done in a single pass. This
makes it difficult to compress DWARF sections because that must be
done after relocations are applied, which must happen after virtual
address layout, but we can't layout the file until we've compressed
the DWARF sections.
Fix this by separating the two layout steps. In the process, we can
also unify the copy-pasted code in Link.address to compute file
offsets. Currently, each instance of this is slightly different, but
there's no reason for it to be. For example, we don't perform
PEFILEALIGN alignment on Segrodata or Selreltodata even when HeadType
== Hwindows, but it turns out it doesn't matter whether you do or
don't because these segments simply don't exist on Windows. Hence, in
the unified code path, we do this alignment for all segments.
Likewise, there are two ways of computing Fileoff:
seg.Vaddr - prev.Vaddr + prev.Fileoff
and
prev.Fileoff + uint64(Rnd(int64(prev.Filelen), int64(*FlagRound)))
At the moment, these always have the same value, but the latter will
continue to work after we start compressing sections on disk.
Tested by comparing test binaries for all packages in std before and
after this change for GOOS={linux,windows,darwin,plan9}. All binaries
are identical.
For #11799.
Change-Id: If09f28771bb4d78dd392fd58b8d7c9d5f22b0b9f
Reviewed-on: https://go-review.googlesource.com/111682
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
If go toolchain is not built with GOEXPERIMENT=fieldtrack,
skip fieldtrack pass in the linker as it does full symtab traversal.
For linking "hello world" example from net/http:
name old time/op new time/op delta
Linker-4 530ms ± 2% 525ms ± 2% -1.03% (p=0.028 n=17+19)
Fixes#20318
Updates #14624
Change-Id: I99336513db77d13f95f47d27339d76f01c42a5da
Reviewed-on: https://go-review.googlesource.com/113635
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
s/Thearch/thearch/
This reduces the amount of exported global variables,
which in turn could make it easier to refactor them later.
Also updated somewhat vague comment about ld.Thearch.
There is no need for Thearch to be exported as Archinit is
called by ld.Main.
Updates #22095
Change-Id: I266b291f6eac0165f70c51964738206e066cea08
Reviewed-on: https://go-review.googlesource.com/103878
Run-TryBot: Iskander Sharipov <iskander.sharipov@intel.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
Suppose you build the Go toolchain in directory A,
move the whole thing to directory B, and then use
it from B to build a new program hello.exe, and then
run hello.exe, and hello.exe crashes with a stack
trace into the standard library.
Long ago, you'd have seen hello.exe print file names
in the A directory tree, even though the files had moved
to the B directory tree. About two years ago we changed
the compiler to write down these files with the name
"$GOROOT" (that literal string) instead of A, so that the
final link from B could replace "$GOROOT" with B,
so that hello.exe's crash would show the correct source
file paths in the stack trace. (golang.org/cl/18200)
Now suppose that you do the same thing but hello.exe
doesn't crash: it prints fmt.Println(runtime.GOROOT()).
And you run hello.exe after clearing $GOROOT from the
environment.
Long ago, you'd have seen hello.exe print A instead of B.
Before this CL, you'd still see hello.exe print A instead of B.
This case is the one instance where a moved toolchain
still divulges its origin. Not anymore. After this CL, hello.exe
will print B, because the linker sets runtime/internal/sys.DefaultGoroot
with the effective GOROOT from link time.
This makes the default result of runtime.GOROOT once again
match the file names recorded in the binary, after two years
of divergence.
With that cleared up, we can reintroduce GOROOT into the
link action ID and also reenable TestExecutableGOROOT/RelocatedExe.
When $GOROOT_FINAL is set during link, it is used
in preference to $GOROOT, as always, but it was easier
to explain the behavior above without introducing that
complication.
Fixes#22155.
Fixes#20284.
Fixes#22475.
Change-Id: Ifdaeb77fd4678fdb337cf59ee25b2cd873ec1016
Reviewed-on: https://go-review.googlesource.com/86835
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@golang.org>
Cherry Zhang
Jeremy Faller
Than McIntosh
Austin Clements
nikita-vanyasin
Russ Cox
Alex Tokarev
Clément Chigot
Alex Brainman
Alessandro Arzilli
Heschi Kreinick
isharipo
quasilyte