To support the experimentalUseInvalidMetadata mode, we keep around
invalid metadata. This can help gopls features work when, for example,
the go.mod file is broken. It is debatable whether this feature is worth
supporting (see golang/go#54180), but in the meantime there is a very
negative side-effect when module paths are changed in the go.mod file:
we keep around a bunch of workspace packages with a stale module path.
As a result we can be left with a lots of extra type-checked packages
in memory, and many inaccurate diagnostics.
Fix this by skipping packages with invalid metadata when computing
workspace packages. While we may want to use invalid metadata when
finding the best package for a file, it does not make sense to re-
type-check and diagnose all those stale packages.
Fixesgolang/go#43186
Change-Id: Id73b47ea138ec80a9de63b03dae41d4e509b8d5a
Reviewed-on: https://go-review.googlesource.com/c/tools/+/420956
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Suzy Mueller <suzmue@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Each regtest does a significant amount of extra work re-doing things
like parsing and type-checking the runtime package. We can share this
work across regtests by using a shared cache, significantly speeding
them up at the cost of potentially hiding bugs related to timing.
Sharing this work still retains most of the benefit of the regtests, so
implement this in the default mode (formerly called "singleton" and now
renamed to "default"). In a subsequent CL, modes will be cleaned up so
that "default" is the only mode that runs with -short.
Making this change actually revealed a caching bug: our cached package
stores error messages extracted from go/packages errors, but does not
include these errors in the cache key. Fix this by hashing all metadata
errors into the package cache key.
Updates golang/go#39384
Change-Id: I37ab9604149d34c9a79fc02b0e1bc23fcb17c454
Reviewed-on: https://go-review.googlesource.com/c/tools/+/417587
TryBot-Result: Gopher Robot <gobot@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
This should have been in CL 417116.
Also:
- (related to CL 417415), rename packageHandle.check to await
to indicate its blocking nature.
- rename typeCheck to typeCheckImpl, following the pattern.
- move "prefetch" parallel loop into typeCheckImpl.
- add some comments.
Change-Id: Iea2c8e1f1f74fb65afd0759b493509147d87a4bb
Reviewed-on: https://go-review.googlesource.com/c/tools/+/417581
Run-TryBot: Alan Donovan <adonovan@google.com>
Auto-Submit: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Also:
- add test of NewHandle
- update package doc and other doc comments
- factor Store.Handle with NewHandle
- declare Handle before Store
Change-Id: I4bcea2c9debf1e77f973ef7ea9dbe2fd7a373996
Reviewed-on: https://go-review.googlesource.com/c/tools/+/417417
Auto-Submit: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Run-TryBot: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
The trimming optimization deletes parts of the syntax tree
that don't affect the type checking of package-level declarations.
It used to remove unexported struct fields, but this had
observable consequences: it would affect the offset of later
fields, and the size and aligment of structs, causing the
'fieldalignment' analyzer to report incorrect findings.
Also, it required a complex workaround in the UI element
for hovering over a type to account for the missing parts.
This change restores unexported fields.
The logic of recordFieldsUses has been inlined and specialized
for each case (params+results, struct fields, interface
methods) as they are more different than alike.
BenchmarkMemStats on k8s shows +4% HeapAlloc:
a lot, but a small part of the 32% saving of the trimming
optimization as a whole.
Also:
- trimAST: delete func bodies without visiting them.
- minor clarifications.
Updates golang/go#51016
Change-Id: Ifae15564a8fb86af3ea186af351a2a92eb9deb22
Reviewed-on: https://go-review.googlesource.com/c/tools/+/415503
gopls-CI: kokoro <noreply+kokoro@google.com>
Run-TryBot: Alan Donovan <adonovan@google.com>
Auto-Submit: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
All Get/Set operations on the maps now happen within a single
function (buildPackageKey, actionHandle).
No behavior change.
Change-Id: I347dfda578c28657a28538e228ecfb6f0871b94b
Reviewed-on: https://go-review.googlesource.com/c/tools/+/417415
Run-TryBot: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Auto-Submit: Alan Donovan <adonovan@google.com>
As with mod tidy in CL 417116, we can rely on invalidation
rather than cache keys to avoid reusing stale results, and
there's no need to save these handles in the global store.
Change-Id: I3763c01fa21c6114248c1d541e3c168fc6a128c9
Reviewed-on: https://go-review.googlesource.com/c/tools/+/417416
Reviewed-by: Robert Findley <rfindley@google.com>
Auto-Submit: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Run-TryBot: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Previously, the modtidy operation used a persistent map
of handles in the central store that cached the result
of a parsing the go.mod file after running 'go mod tidy'.
The key was complex, including the session, view, imports
of all dependencies, and the names of all unsaved overlays.
The fine-grained key prevented spurious cache hits for
invalid inputs by (we suspect) preventing nearly all cache hits.
The existing snapshot invalidation mechanism should be
sufficient to solve this problem, as the map entry is evicted
whenever the metadata or overlays change. So, this change
avoids keeping handles in the central store, so they are
never shared across views.
Also, modtidy exploited the fact that a packageHandle
used to include a copy of all the Go source files
of each package, to avoid having to read the files
itself. As a result it would entail lots of unnecessary
work building package handles and reading dependencies
when it has no business even thinking about type checking.
This change:
- extracts the logic to read Metadata.{GoFiles,CompiledGo}Files
so that it can be shared by modtidy and buildPackageHandle.
- packageHandle.imports has moved into mod_tidy.
One call (to compute the hash key) has gone away,
as have various other hashing operations.
- removes the packagesMap typed persistent.Map wrapper.
- analysis: check cache before calling buildPackageHandle.
- decouple Handle from Store so that unstored handles may
be used.
- adds various TODO comments for further simplification.
Change-Id: Ibdc086ca76d6483b094ef48aac5b1dd0cdd04973
Reviewed-on: https://go-review.googlesource.com/c/tools/+/417116
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
Run-TryBot: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Auto-Submit: Alan Donovan <adonovan@google.com>
...now that I have understood why they are not
in fact inessential.
Change-Id: I1ab881a7d24cd71ee183bc8c6a1a058dbda641e2
Reviewed-on: https://go-review.googlesource.com/c/tools/+/416077
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Alan Donovan <adonovan@google.com>
Run-TryBot: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Now that the lifetime of all handles in the store is
determined by reference counting, we no longer need
the generation feature.
The Arg interface, renamed RefCounted, is now optional,
and causes the lifetime of the argument to be extended
for the duration of the Function call. This is important
when the Get(ctx) context is cancelled, causing the
function call to outlive Get: if Get's reference to
the argument was borrowed, it needs to increase the
refcount to prevent premature destruction.
Also:
- add missing snapshot.release() call in
importsState.populateProcessEnv.
- remove the --memoize_panic_on_destroyed flag.
Change-Id: I0b3d37c16f8b3f550bb10120c066b628c3db244b
Reviewed-on: https://go-review.googlesource.com/c/tools/+/416076
Run-TryBot: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Alan Donovan <adonovan@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
This change simplifes the ParseGo interface to
make it consistent with the other handle+map operations:
- ParseGoImpl is the basic parser.
- The 'fixed' bool result is a field of ParsedGoFile.
- ParseGo is the caching wrapper.
The map accessors have been inlined into it.
- goFiles (renamed parsedGoFiles) is now just a bare
persistent.Map.
- parseGoHandle is replaced by *memoize.Handle
- the operations of "make a handle" and "wait for it"
are no longer separate (since clients never want
one without the other).
- cachedPGF and peekOrParse have been combined into
peekParseGoLocked.
Change-Id: If01a6aaa7e6a8d78cb89c305e5279738e8e7bb55
Reviewed-on: https://go-review.googlesource.com/c/tools/+/416223
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Run-TryBot: Alan Donovan <adonovan@google.com>
If metadata is refreshed during the execution of buildPackageHandle, we
should not store the resulting package handle in the snapshot, as it
breaks the invariant that computed packages match the currently loaded
metadata.
This strictness revealed another bug: because of our fine-grained
locking in snapshot.load, it is possible that we set valid metadata
multiple times, leading to unnecessary invalidation and potential
further races to package handles. Fix this by building all metadata when
processing the go/packages result, and only filtering updates after
grabbing the lock.
For golang/go#53733
Change-Id: Ib63ae4fbd97d0d25d45fe04f9bcd835996db41da
Reviewed-on: https://go-review.googlesource.com/c/tools/+/416224
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
span.NewRange now accepts a *token.File and two token.Pos.
It is the caller's responsibility to look up the File
in the FileSet, if necessary (it usually isn't), and to
ensure the Pos values are valid. Ditto NewMappedRange.
This reduces the creep of Snapshot into functions that
have no need to know about it. Also the bug.Report call
in NewRange has been pushed up into the caller and
logically eliminated in all but one case.
I think we should aim for the invariant that functions that
operate on a single file should accept a *token.File, not
a FileSet; only functions that operate on sets of files
(e.g. type checking, analysis) should use a FileSet.
This is not always possible: some public functions
accept a FileSet only to re-lookup a single file already
known to the caller; if necessary we could provide token.File
variants of these.
This may ultimately allow us to create a new FileSet per
call to the parser, so that Files and FileSets are in
1:1 correspondance and there is no global FileSet.
(It currently grows without bound, on the order of kilobytes
per keystroke.) FileSets containing multiple files,
needed when we interact with the type checker and analysis,
may be temporarily synthesized on demand from a set of Files,
analogous to mmap'ing a few files into a blank address space.
Also:
- replace File.Position(pos).Line by File.Line(pos)
- replace pos == token.NoPos by pos.IsValid()
- avoid fishy token.Pos conversions in link.go
- other minor simplifications
Change-Id: Ia3119e0ac7e193801fbafa81c8f48acfa14e9ae4
Reviewed-on: https://go-review.googlesource.com/c/tools/+/409935
Auto-Submit: Alan Donovan <adonovan@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Run-TryBot: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
This speeds up workspace initialization time in DegradeClosed memory
mode from 3m to 1m by avoiding unnecessary recomputation of results.
Change-Id: Ie5df82952d50ab42125defd148136329f0d50a48
Reviewed-on: https://go-review.googlesource.com/c/tools/+/413658
Reviewed-by: Alan Donovan <adonovan@google.com>
Reviewed-by: David Chase <drchase@google.com>
...so that each client doesn't have to.
Change-Id: I039c493031c5c90c4479741cf6f7572dad480808
Reviewed-on: https://go-review.googlesource.com/c/tools/+/415502
Run-TryBot: Alan Donovan <adonovan@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Auto-Submit: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Before, buildPackageHandle and buildKey were mutually recursive.
Together they performed a sequential recursion over Metadata.Deps,
calling GetFile and parseGoHandle for every file, and then
finally (in postorder) binding a Handle for the type checking step.
This change inlines buildKey to make the recursion more obvious,
performs the recursion over dependencies first, followed by
the reading of Go source files for this package, in parallel.
(The IWL benchmark reports improvement but its variance is so
high I'm not sure I trust it.) Other opportunities for parallelism
are pointed out in new comments.
The Bind operation for typechecking calls dep.check for each
dependency in a separate goroutine. It no longer waits for
each one since it is only prefetching the information
that will be required during import processing, which will
block until the information becomes available.
Before, both reading and parsing appear to occur twice:
once in buildPackageHandle and once in doTypeCheck.
(Perhaps the second was a cache hits, but there's no need
to rely on a cache.)
Now, only file reading (GetFile) occurs in buildPackageHandle,
since that's all that's needed for the packageKey.
And parsing only occurs in doTypeCheck. The source.FileHandles
are plumbed through as parameters.
Also:
- move parseGoHandles to a local function, since it exists only
for buildPackageKey. It no longer parses, it only reads.
- lots of TODO comments for possible optimizations,
and typical measured times of various operations.
- remove obsolete comment re: Bind and finalizers.
Change-Id: Iad049884607b73eaa6701bdf7771f96b042142d5
Reviewed-on: https://go-review.googlesource.com/c/tools/+/411913
Run-TryBot: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Auto-Submit: Alan Donovan <adonovan@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
This on average reduces latency from 25ms to 12ms on internal codebase.
Updates golang/go#45686
Change-Id: I49c8f09f8e54b7b486d7ff7eb8f4ba9f0d90b278
Reviewed-on: https://go-review.googlesource.com/c/tools/+/413655
gopls-CI: kokoro <noreply+kokoro@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
This reverts commit 654a14b527.
Reason for revert: my flawed understanding of the concurrency
Change-Id: I31a35267323bb1ff4dff1d9244d3ce69c36cdda4
Reviewed-on: https://go-review.googlesource.com/c/tools/+/412694
Run-TryBot: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
This change reduces the sizes of the critical sections
in traces.ProcessEvent and Generation.Bind, in particular
moving allocations ahead of Lock. This reduces the contention
according to the trace profiler.
See https://go-review.googlesource.com/c/go/+/411909 for
another reduction in contention. The largest remaining
contention is Handle.Get, which thousands of goroutines
wait for because we initiate typechecking top down.
Also, add a couple of possible optimization TODO comments,
and delete a stale comment re: Bind.
Change-Id: I995a0bb46e8c9bf0c23492fb62b56f4539bc32f8
Reviewed-on: https://go-review.googlesource.com/c/tools/+/411910
Run-TryBot: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
I had hoped to see a reduction in total allocation, but it does not
appear to be significant according to the included crude benchmark.
Nonetheless this is a slight code clarity improvement.
Change-Id: I94a503b377dd1146eb371ff11222a351cb5a43b7
Reviewed-on: https://go-review.googlesource.com/c/tools/+/411655
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
Package loading (at least via go list) fails when the import header
doesn't parse, so we need to invalidate metadata upon a state change
from non parsing->parsing. Refactor metadata invalidation to implement
this feature, add additional documentation, and avoid unnecessary work.
This change revealed a latent bug (via TestDeleteDirectory): when
statting a deleted directory failed, we could fail to invalidate any
package IDs, even those we already knew about. This bug was masked by
the somewhat complicated semantics of pkgNameChanged. The semantics of
pkgNameChanged are simplified to encapsulate any change to a
package->file association.
Also refactor the parsing API somewhat, and add a bug report if we don't
get a ParseGoFile while inspecting for metadata changes.
Update most regtests to panic upon receiving bug reports.
Fixesgolang/go#52981
Change-Id: I1838963ecc9c01e316f887aa9d8f1260662281ab
Reviewed-on: https://go-review.googlesource.com/c/tools/+/407501
Reviewed-by: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Robert Findley <rfindley@google.com>
As of golang/go#50827, gopls no longer supports building at 1.12, and so
usage of golang.org/x/xerrors can be replaced with the native support for
error wrapping introduced in Go 1.13.
Remove this usage as a step toward eliminating the xerrors dependency
from x/tools.
For golang/go#52442
Change-Id: Ibf459cc72402a30a6c2735dc620f76ed8a5e2525
Reviewed-on: https://go-review.googlesource.com/c/tools/+/401097
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Ian Lance Taylor <iant@google.com>
The go directive affects type checking errors so
it should be included in the package key.
I manually tested that this fixes the issue in golang/go#51325Fixesgolang/go#51325
Change-Id: I109859ae65e7e4d5fdefc63a0a7a838117ee02cf
Reviewed-on: https://go-review.googlesource.com/c/tools/+/387914
Trust: Suzy Mueller <suzmue@golang.org>
Run-TryBot: Suzy Mueller <suzmue@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
Set the language version from the controlling go.mod file's go version,
if any. Also verify that we properly surface a diagnostic if the version
is invalid.
I didn't add any quick fixes.
Fixesgolang/go#50688.
Change-Id: Ic472502d1224a1decb5b989d51110b837020e998
Reviewed-on: https://go-review.googlesource.com/c/tools/+/383394
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
This CL updates the internal/typeparams package with the new API from CL
349629, switching Info.Inferred to Info.Instances.
Change-Id: I8f98dc39c844ef8891863c08b747f63924ab1c53
Reviewed-on: https://go-review.googlesource.com/c/tools/+/351250
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@golang.org>
In preparation for moving metadata related functionality to a separate
package, move around some types and export some symbols. This is purely
to reduce diffs in subsequent CLs, and contains no functional changes.
Change-Id: I24d4fbd71df78e4d7a84f6598cdf820b41d542a2
Reviewed-on: https://go-review.googlesource.com/c/tools/+/340729
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
In my testing, the gopls degraded memory mode (currently set via
"memoryMode": "DegradeClosed") did not save as much memory as expected
due to still type checking all packages in the workspace (even if in
ParseExported mode). It is also annoying to get incomplete results from
references and renaming.
I think we can (and should) fix both problems: don't even consider
packages that aren't 'reachable' via open files, but fully type check
the reverse transitive closure of the packages you're working on. This
CL does exactly that, by swapping out the concept of 'workspace
packages' with 'active packages'.
In testing, this decreased my memory footprint while working on std by
3-4x when compared to normal mode, and 2x when compared to the previous
implementation of DegradeClosed.
It still needs more testing before we move this option out of
experimental.
For golang/go#46902
Change-Id: I1e319d0b1607d344d27e797ce32de057d7a583f9
Reviewed-on: https://go-review.googlesource.com/c/tools/+/336410
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
When we open a file in a package, independent of whether it is in the
workspace, we type check in ParseFull mode. However, several other
code paths don't find this better parse mode.
We need a better abstraction, but for now improve a couple code paths
specifically for the purpose of fixing Hover content.
Updates golang/go#46158
Updates golang/go#46902
Change-Id: I34c0432fdba406d569ea963ab4366336068767a2
Reviewed-on: https://go-review.googlesource.com/c/tools/+/333689
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Use filepath.SplitList rather than always splitting GOPATH on ':'.
Fixesgolang/go#46805
Change-Id: I27324afba96b550f75cc272b6c7f701b28825d39
Reviewed-on: https://go-review.googlesource.com/c/tools/+/328969
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
As an experiment, this CL introduces the first gopls feature that is
specific to generics: enriching function hover information with inferred
types. This is done with no additional gating on build constraints by
using the new internal/typeparams package.
The marker tests are updated to allow tests that rely on type parameters
being enabled.
Change-Id: Ic627d64b61a6211389196814edd0abe1484491eb
Reviewed-on: https://go-review.googlesource.com/c/tools/+/317452
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Retrying CL 271477, this time with parts of CL 322650 incorporated.
This CL moves to a model where we don't automatically delete invalidated
metadata, but rather preserve it and mark it invalid. This way, we can
continue to use invalid metadata for all features even if there is an
issue with the user's workspace.
To keep track of the metadata's validity, we add an invalid flag to
track the status of the metadata. We still reload at the same rate--the
next CL changes the way we reload data.
We also add a configuration to opt-in (currently, this is off by
default).
In some cases, like switches between GOPATH and module modes, and when a
file is deleted, the metadata *must* be deleted outright.
Also, handle an empty GOMODCACHE in the directory filters (from a
previous CL).
Updates golang/go#42266
Change-Id: Idc778dc92cfcf1e4d14116c79754bcca0229e63d
Reviewed-on: https://go-review.googlesource.com/c/tools/+/324394
Trust: Rebecca Stambler <rstambler@golang.org>
Run-TryBot: Rebecca Stambler <rstambler@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
This reverts commit 46e69bf3b2.
Reason for revert: Still has bugs associated with it and want to do a release
Change-Id: Ifa80bde147aa23aa4029a157d5dbaf6479d53024
Reviewed-on: https://go-review.googlesource.com/c/tools/+/324290
Trust: Rebecca Stambler <rstambler@golang.org>
Run-TryBot: Rebecca Stambler <rstambler@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
This CL adds two new commands that let a client request a list of importable packages relative to a Go file and then select which import a programmer would like to add to said file.
Updates golang/go#43351
Change-Id: If12518874a92ed4167bdd711a92e03ee21c7b949
Reviewed-on: https://go-review.googlesource.com/c/tools/+/281412
Trust: Rebecca Stambler <rstambler@golang.org>
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Rebecca Stambler <rstambler@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
This CL moves to a model where we don't automatically delete invalidated
metadata, but rather preserve it and mark it invalid. This way, we can
continue to use invalid metadata for all features even if there is an
issue with the user's workspace.
To keep track of the metadata's validity, we add an invalid flag to
track the status of the metadata. We still reload at the same rate--the
next CL changes the way we reload data.
We also add a configuration to opt-in (currently, this is off by
default).
In some cases, like switches between GOPATH and module modes, and when a
file is deleted, the metadata *must* be deleted outright.
Updates golang/go#42266
Change-Id: Iff5e10b641fdb4be270af0cd887a10ee97ac1a19
Reviewed-on: https://go-review.googlesource.com/c/tools/+/271477
Trust: Rebecca Stambler <rstambler@golang.org>
Run-TryBot: Rebecca Stambler <rstambler@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
I wrote this code as if there was going to be a final error check after
all the type checking attempts, but ended up using the result inside the
attempts, errors would likely have resulted in panics. Just do normal,
non-clever error checking.
Change-Id: I665f34f7e6d1a2c3465543cbdc39a723a22a1095
Reviewed-on: https://go-review.googlesource.com/c/tools/+/319371
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Despite the name, ParseExported only hollowed out declarations -- it
didn't actually drop any from the AST. This leaves a fair amount of
unexported crud behind. Unfortunately, there are a *lot* of ways to
expose an unexported declaration from an exported one, and it can be
done across files. Because of that, discarding unexported declarations
requires a lot of work.
This CL implements a decent attempt at pruning as much as possible from
the AST in ParseExported mode.
First, we analyze the AST of all the files in the package for exported
uses of unexported identifiers, iterating to a fixed point. Then, we
type check those ASTs. If there are missing identifiers (probably due to
a bug in the dependency analysis) we use those errors to re-parse. After
that we give up and fall back to the older, less effective trimming. The
pkg type changes slightly to accomodate the new control flow.
We have to analyze all the files at once because an unexported type
might be exposed in another file. Unfortunately, that means we can't
parse a single file at a time any more -- the result of parsing a file
depends on the result of parsing its siblings. To avoid cache
corruption, we have to do the parsing directly in type checking,
uncached.
This, in turn, required changes to the PosTo* functions. Previously,
they operated just on files, but a file name is no longer sufficient to
get a ParseExported AST. Change them to work on Packages instead. I
squeezed in a bit of refactoring while I was touching them.
Change-Id: I61249144ffa43ad645ed38d79e873e3998b0f38d
Reviewed-on: https://go-review.googlesource.com/c/tools/+/312471
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
We still hear from users for whom gopls uses too much memory. My efforts
to reduce memory usage while maintaining functionality are proving
fruitless, so perhaps it's time to accept some functionality loss.
DegradeClosed MemoryMode typechecks all packages in ParseExported mode
unless they have a file open. This should dramatically reduce memory
usage in monorepo-style scenarious, where a ton of packages are in the
workspace and the user might plausibly want to edit any of them.
(Otherwise they should consider using directory filters.)
The cost is that features that work across multiple packages...won't.
Find references, for example, will only find uses in open packages or in
the exported declarations of closed packages.
The current implementation is a bit leaky; we keep the ParseFull
packages in memory even once all their files are closed. This is related
to a general failure on our part to drop unused packages from the
snapshot, so I'm not going to try to fix it here.
Updates golang/go#45457, golang/go#45363.
Change-Id: I38b2aeeff81a1118024aed16a3b75e18f17893e2
Reviewed-on: https://go-review.googlesource.com/c/tools/+/310170
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
This broke staticcheck and x/tools/refactor, most notably used for our
rename support. Doesn't look like a winner. Roll it back :(
Updates golang/go#45457.
Change-Id: I30d5aa160fd9319329d36b2a534ee3c756090726
Reviewed-on: https://go-review.googlesource.com/c/tools/+/311549
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
As predicted in CL 308730, there are some analyzers that need this: all
of staticcheck. (I probably should've seen that coming.) For now, don't
do the trimming when staticcheck is on. Since staticcheck ~doubles
memory footprint, those users probably don't care much.
Change-Id: I7cfd96b3654e3617d28d62015928a45a85407d13
Reviewed-on: https://go-review.googlesource.com/c/tools/+/311376
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
The type checker emits a TypeAndValue entry for (among other things)
every constant in a Go file. Normally, that cost is moderate. However,
in the case of a large slice literal, it can get out of control very
quickly. Imagine a code generator that creates a 2KB byte slice literal;
that's 2K TypeAndValue entries, each of which is considerably larger
than the 1-3 bytes for the source text.
Unfortunately, there are a number of such code generators. Notably,
there are such slice literals in proto code, e.g.
https://github.com/grpc/grpc-go/blob/master/examples/route_guide/routeguide/route_guide.pb.go#L360
This CL changes the type checking code to remove the TypeAndValue
entries for slice literals of basic types after the checker returns.
In the extreme case of https://github.com/googleapis/go-genproto, which
is nothing but protos, I see a ~40% drop in heap usage.
I believe this change is generally safe, but there's no way to guarantee
it. I don't think any editor features need to know the type or value of
an arbitrary slice element, but it is just barely possible that an
analyzer does.
Change-Id: Iee1af2369f994597a42fd1dcbf8af20faa43410e
Reviewed-on: https://go-review.googlesource.com/c/tools/+/308730
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
According to its comment, reloadOrphanedFiles is intended to work around
overlay bugs. But as of 1.16, there are no overlay bugs that we know of.
So what is it still doing?
Apparently, quite a bit, not much of it useful. Clean up as much as
possible.
- Files with no valid package declaration are ignored by the go command.
There's no point trying to reload them; stop.
- During metadata invalidation, we clear out all IDs for a file, even if
only one of its IDs is invalidated, e.g. when a test package is removed.
That leaves valid metadata for the non-test, so we don't refresh it in
the workspace reload, and only catch it as an orphan. It seems to me we
should only remove the invalidated ID.
- If the client incorrectly sends us a didOpen for a non-Go file, we
will attempt to load it as an orphaned file. Fix the regtest that did
that.
- TestEmptyGOPATHXTest_40825 set up an invalid GOPATH: you can't work in
GOPATH/src. However, it exists to test code that no longer exists, so
just delete it.
After this change, almost none of the regression tests trigger orphaned
file reloading. It's difficult/impractical to rule it out entirely
because some of them only appear racily. Since I intend to remove the
code path, I'm not too worried about more creeping in before I'm done.
The only useful case is multiple ad-hoc packages. Because we only
allow one "command-line-arguments" package in the workspace, if you
switch between two the old one becomes orphaned. I hope to work on that
soon.
Change-Id: Ia355cf104280ce51f6189c6638e8da8f4aef2ace
Reviewed-on: https://go-review.googlesource.com/c/tools/+/302089
Trust: Heschi Kreinick <heschi@google.com>
Trust: Rebecca Stambler <rstambler@golang.org>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Generating go get quick fixes in a single place only made it harder to
get them right. Do it during diagnostics generation, as is the new
norm.
Also improve the user experience. When we fail to import a package
because one of its dependencies is missing, it makes more sense to run
go get on the package we tried to import, not the one that's missing:
that will download all of its missing dependencies if there happen to be
more.
Change-Id: Ib6a8140bccfafcb9f966d25639799dd4c7347c3d
Reviewed-on: https://go-review.googlesource.com/c/tools/+/300072
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
In CL 295413 we fixed the handling of related type checker diagnostics
to correctly identify the primary and secondary errors at a position.
However, on clients that don't support diagnostic related information,
this can lead to confusing primary diagnostics.
Add handling for clients that don't support related information, to
embed the secondary error in the primary error.
Fixesgolang/go#44735
Change-Id: I3d2470d2a4044661e6ed31ac9ffd2f9ff27f7d4b
Reviewed-on: https://go-review.googlesource.com/c/tools/+/297875
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
CL 295414 suppressed all type checking errors in the presence of parser
errors, but this was a bit overzealous. go/types attempts to suppress
follow-on errors from bad syntax, so as long as we haven't had to modify
the AST and the current file parses, type checking errors can still
provide a decent signal to the user
Fixesgolang/go#44736
Change-Id: Icd89404fbae663b8f934a38908eaaa87c561b64a
Reviewed-on: https://go-review.googlesource.com/c/tools/+/297872
Trust: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
In practice, the only code shared among the various switch cases was the
call to spanToRange, which definitely doesn't justify the giant
function. Split it out into per-type functions.
I removed the unused case for types.Error, and a bit of error checking I
believe to be redundant. I don't intend any functional changes.
At this point it might be worth considering moving the functions into
other files, but I don't think it matters that much.
Change-Id: I05b4d86dd37a9ff1887a116183c915c225faf3a7
Reviewed-on: https://go-review.googlesource.com/c/tools/+/297129
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Hopefully improve some of the details around parsing that have always
confused me.
- parser.ParseFile will never return an error other than
scanner.ErrorList. Encode that in ParsedGoFile.
- parser.ParseFile will never return a nil file. Eliminate the code
path that handled that.
- Explain why we might fail to find a token.File.
- Trying to fix errors appears quite expensive even if there aren't any
to fix. Don't waste the time.
Change-Id: I87e082ed52c98a438bc7fd3b29e1a486f32fb347
Reviewed-on: https://go-review.googlesource.com/c/tools/+/297069
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
In typeCheckDiagnostics, we have logic to suppress go list and type
checking errors based on the presence of other errors. Nobody seems to
know why the logic is exactly what it is, and suppressing list errors
means that bad //go:embed patterns don't show up.
Intuitively, it makes sense to me that we don't want to type check if
listing or parsing fails: list errors mean that whole files may be
missing, and parsing errors may wipe out arbitrary chunks of files.
There's little point reporting errors from type checking that. However,
list errors and parse errors should be mostly orthogonal: go list
parses very little of the file and in practice only reports errors
parsing the package statement. So, at least for now, we report both
parse and list errors, and stop there if there are any.
Finally, move the suppression logic to the actual typeCheck function
that generates the diagnostics. typeCheckDiagnostics is the primary
consumer, but I think it's better to do the suppression at the source.
Because we are now showing list errors, and they are prone to getting
stuck due to bad overlay support, a couple of tests now require 1.16.
Change-Id: I7801e44c4d3da30bda61e0c1710a2f52de6215f5
Reviewed-on: https://go-review.googlesource.com/c/tools/+/295414
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Robert Findley <rfindley@google.com>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Due to misuse of the expandErrors function, when we encountered a
continuing error, we would group all the related errors into a single
one, then return both it and all its secondaries individually. That
makes for a strange user experience where you get one complete error and
N-1 incomplete ones.
Instead, create a copy of the complete error for each secondary error,
moving the copy to the location of the secondary error and adding a bit
to the text to (hopefully) clarify what's going on.
Change-Id: I87cf0e3b2cea98317f650d16a78476edf108a934
Reviewed-on: https://go-review.googlesource.com/c/tools/+/295413
Trust: Heschi Kreinick <heschi@google.com>
Run-TryBot: Heschi Kreinick <heschi@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Rebecca Stambler <rstambler@golang.org>
Robert Findley
Alan Donovan
Ruslan Nigmatullin
Suzy Mueller
Heschi Kreinick
Rebecca Stambler
Marwan Sulaiman