x/tools/gopls: long initial workspace load durations for workspace with many packages

[rma-stripe]

gopls version

Build info
----------
golang.org/x/tools/gopls (devel)
    golang.org/x/tools/gopls@(devel)
    github.com/BurntSushi/toml@v1.2.1 h1:9F2/+DoOYIOksmaJFPw1tGFy1eDnIJXg+UHjuD8lTak=
    github.com/google/go-cmp@v0.6.0 h1:ofyhxvXcZhMsU5ulbFiLKl/XBFqE1GSq7atu8tAmTRI=
    golang.org/x/exp/typeparams@v0.0.0-20221212164502-fae10dda9338 h1:2O2DON6y3XMJiQRAS1UWU+54aec2uopH3x7MAiqGW6Y=
    golang.org/x/mod@v0.21.0 h1:vvrHzRwRfVKSiLrG+d4FMl/Qi4ukBCE6kZlTUkDYRT0=
    golang.org/x/sync@v0.8.0 h1:3NFvSEYkUoMifnESzZl15y791HH1qU2xm6eCJU5ZPXQ=
    golang.org/x/telemetry@v0.0.0-20240829154258-f29ab539cc98 h1:Wm3cG5X6sZ0RSVRc/H1/sciC4AT6HAKgLCSH2lbpR/c=
    golang.org/x/text@v0.18.0 h1:XvMDiNzPAl0jr17s6W9lcaIhGUfUORdGCNsuLmPG224=
    golang.org/x/tools@v0.21.1-0.20240508182429-e35e4ccd0d2d => ../
    golang.org/x/vuln@v1.0.4 h1:SP0mPeg2PmGCu03V+61EcQiOjmpri2XijexKdzv8Z1I=
    honnef.co/go/tools@v0.4.7 h1:9MDAWxMoSnB6QoSqiVr7P5mtkT9pOc1kSxchzPCnqJs=
    mvdan.cc/gofumpt@v0.7.0 h1:bg91ttqXmi9y2xawvkuMXyvAA/1ZGJqYAEGjXuP0JXU=
    mvdan.cc/xurls/v2@v2.5.0 h1:lyBNOm8Wo71UknhUs4QTFUNNMyxy2JEIaKKo0RWOh+8=
go: go1.23.1

go env

$ bin/go env
GO111MODULE=''
GOARCH='arm64'
GOBIN=''
GOCACHE='/Users/rma/Library/Caches/go-build'
GOENV='/Users/rma/Library/Application Support/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='arm64'
GOHOSTOS='darwin'
GOINSECURE=''
GOMODCACHE='/Users/rma/go/pkg/mod'
GONOPROXY=<REDACTED>
GONOSUMDB=<REDACTED>
GOOS='darwin'
GOPATH='/Users/rma/go'
GOPRIVATE=<REDACTED>
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/Users/rma/.cache/gocode/sdk/1.22'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/Users/rma/.cache/gocode/sdk/1.22/pkg/tool/darwin_arm64'
GOVCS=''
GOVERSION='go1.22.6'
GCCGO='gccgo'
AR='ar'
CC='clang'
CXX='clang++'
CGO_ENABLED='1'
GOMOD='/Users/rma/stripe/gocode/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -ffile-prefix-map=/var/folders/wc/_g6hd_8d3dnb87c04x67w0l40000gn/T/go-build681359692=/tmp/go-build -gno-record-gcc-switches -fno-common'

What did you do?

My organization within the company is responsible for maintaining a large Go monorepo (~20k packages). In this monorepo, all packages fall under 1 Go module.

For the most part, we run a fairly off-the-shelf gopls setup without any customization of note. This is what our stats look like (from bea7373d8a8268c2e3a260c1b8d41f96c4f7489e):

$ gopls stats -anon
Initializing workspace...     done (3m26.109285333s)
Gathering bug reports...      done (1.781858458s)
Querying memstats...          done (1.284304292s)
Querying workspace stats...   done (701.076375ms)
Collecting directory info...  done (7.958301292s)
{
  "DirStats": {
    "Files": 288174,
    "TestdataFiles": 8243,
    "GoFiles": 77688,
    "ModFiles": 20,
    "Dirs": 64532
  },
  "GOARCH": "arm64",
  "GOOS": "darwin",
  "GOPACKAGESDRIVER": "",
  "GOPLSCACHE": "",
  "GoVersion": "go1.23.1",
  "GoplsVersion": "(devel)",
  "InitialWorkspaceLoadDuration": "3m26.109285333s",
  "MemStats": {
    "HeapAlloc": 2970803424,
    "HeapInUse": 4669145088,
    "TotalAlloc": 36820682064
  },
  "WorkspaceStats": {
    "Files": {
      "Total": 72996,
      "Largest": 9189430,
      "Errs": 0
    },
    "Views": [
      {
        "GoCommandVersion": "go1.22.0",
        "AllPackages": {
          "Packages": 23170,
          "LargestPackage": 624,
          "CompiledGoFiles": 89502,
          "Modules": 1031
        },
        "WorkspacePackages": {
          "Packages": 18127,
          "LargestPackage": 354,
          "CompiledGoFiles": 61173,
          "Modules": 1
        },
        "Diagnostics": 95
      }
    ]
  }
}

What did you see happen?

The experience of using gopls is good (i.e. rpcs like jump-to-definition, refactor symbol, etc. are fast) once initial workspace load has occurred, but initial workspace loads are a poor experience for our users. This poor experience is owed to two behaviors:

Firstly, initial workspace loads (IWLs) are slow. This is partially owed to the performance of the expensive upfront go list call itself:

$ time go list -e -json=Name,ImportPath,Error,Dir,GoFiles,IgnoredGoFiles,IgnoredOtherFiles,CFiles,CgoFiles,CXXFiles,MFiles,HFiles,FFiles,SFiles,SwigFiles,SwigCXXFiles,SysoFiles,TestGoFiles,XTestGoFiles,CompiledGoFiles,Export,DepOnly,Imports,ImportMap,TestImports,XTestImports,ForTest,DepsErrors,Module,EmbedFiles -compiled=true -test=true -export=false -deps=true -find=false -pgo=off -- /Users/rma/stripe/gocode/... builtin
[ ...A bunch of redacted `go list` output... ]
go list -e  -compiled=true -test=true -export=false -deps=true -find=false  -  31.41s user 48.00s system 120% cpu 1:06.01 total

…but the majority of the time is spent on calling packages.Load on all of the packages that go list returns.

Secondly, because IWLs are blocking and VSCode waits on code actions before the IDE is able to save files, users are unable to save files/exit their IDEs/etc:

Saving 'main.go': Getting code actions from ''Go', 'ESLint', 'GitHub Copilot Chat'' (configure).

…while IWL is incomplete. This is partially a corollary of the first issue. While it is true that making code actions non-blocking for saves would fix this UX issue, if IWL were fast, this UX issue would not be perceived by users in the first place.

We have dipped our toes in trying to patch this behavior. For example, an early attempt at hacking around this behavior (on a v1.16.1 base) no-ops the initialization routine altogether, relying solely on gopls loading in package data as files are manually opened by the user. As expected, this hack has some large tradeoffs associated with it, because initialization is a load bearing part of gopls: it breaks features like "Rename symbol" and "Find all references", and it occasionally causes the wrong imports to be pulled in. However, buggy as it is, this hack in essence describes what we think the desirable IWL behavior would be:

• The initial snapshot at gopls startup does not await IWL
• On startup, an async IWL routine runs go list and lazy loads package data into the currently "active" snapshot
• While this async routine has not completed, language server RPCs which rely on all workspace packages being loaded (such as "Find all references" are not available)

What did you expect to see?

This issue is a feature request.

• Does the newly proposed workspace initialization approach align with the gopls roadmap?
• Is there a better approach (in the interim) to addressing the slow/blocking behavior of IWLs than the no-op/hack solution linked above?

We would appreciate any tips/guidance in writing an upstream-able patch, or a better hack.

Editor and settings

No response

Logs

No response

[gabyhelp]

Related Issues and Documentation

• x/tools/gopls: high memory consumption when used with a monorepo #37670 (closed)
• x/tools/gopls: add ability to avoid scanning an entire workspace #53278 (closed)
• x/tools/gopls: init project cost long time when open the go project #62715 (closed)
• x/tools/gopls: too many memory usage #37759 (closed)
• x/tools/gopls: crash importing test variant of tools file #66109 (closed)
• x/tools/gopls: feedback is extremely slow with large monorepo #44188 (closed)
• cmd/go: `go list` commands called by `golang.org/x/tools/go/packages.Load()` varies from ~500ms to 3s with no other changes #63863 (closed)
• x/tools/gopls: massive cpu and memory usage #46897 (closed)
• x/tools/gopls: gopls walks entire filesystem of a go module #52993 (closed)
• x/tools/imports: should not scan bazel-out directories #35914

_{(Emoji vote if this was helpful or unhelpful; more detailed feedback welcome in this discussion.)}

[findleyr]

Thanks for filing this thoughtful issue.

I've wanted the ability to bypass the initial workspace load for a while. This is in line with the work in https://go.dev/issue/68002, which I'm working my way up to now.

I expect this to be doable, though there will almost certainly be many edge cases to take care of (for example: producing reasonable errors for workspace-wide misconfiguration).