internal/lsp/cache: use persistent map for storing gofiles in the snapshot #382

euroelessar · 2022-06-09T22:11:30Z

Use treap (https://en.wikipedia.org/wiki/Treap) as a persistent map to avoid copying s.goFiles across generations.
Maintain an additional s.parseKeysByURIMap to avoid scanning s.goFiles on individual file's content invalidation.

This on average reduces didChange latency on internal codebase from 160ms to 150ms.

In a followup the same approach can be used to avoid copying s.files, s.packages, and s.knownSubdirs.

Updates golang/go#45686

gopherbot · 2022-06-09T22:18:20Z

This PR (HEAD: 159eb75) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-09T22:53:23Z

This PR (HEAD: 8c25760) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

euroelessar · 2022-06-09T23:00:03Z

benchmark shows 40% reduction against ad756c7 on linux/amd64 for DidChange

$ go test -v ./gopls/internal/regtest/bench -run=TestBenchmarkDidChange -didchange_dir=$HOME/go/src/k8s.io/kubernetes -didchange_file=pkg/util/hash/hash.go
Before:
BenchmarkStatistics	     456	   3294187 ns/op	 1235424 B/op	    5611 allocs/op
After:
BenchmarkStatistics	     900	   2007147 ns/op	  829696 B/op	    5409 allocs/op

gopherbot · 2022-06-10T14:08:27Z

Message from Alan Donovan:

Patch Set 3:

(5 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-10T18:05:25Z

Message from Alan Donovan:

Patch Set 3:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-10T18:34:22Z

This PR (HEAD: d6691bc) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-10T19:34:35Z

Message from Ruslan Nigmatullin:

Patch Set 4:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-10T19:38:15Z

Message from Ruslan Nigmatullin:

Patch Set 4:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-13T20:47:33Z

This PR (HEAD: fcf814c) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-13T21:02:26Z

This PR (HEAD: 599c2a3) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-13T21:06:34Z

Message from Ruslan Nigmatullin:

Patch Set 6:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-13T21:06:34Z

Message from Ruslan Nigmatullin:

Patch Set 6:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-14T03:54:47Z

This PR (HEAD: 9b6e3e1) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-14T14:35:48Z

Message from Alan Donovan:

Patch Set 9:

(17 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-14T18:27:58Z

Message from Ruslan Nigmatullin:

Patch Set 9:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-15T23:49:53Z

Message from Ruslan Nigmatullin:

Patch Set 9:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-16T00:01:39Z

This PR (HEAD: f0b31fd) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-16T00:05:42Z

This PR (HEAD: 0e5720f) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-16T00:09:43Z

Message from Ruslan Nigmatullin:

Patch Set 11:

(16 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-16T14:05:09Z

Message from Alan Donovan:

Patch Set 12: Code-Review+2

(5 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-16T19:32:33Z

This PR (HEAD: b211d6c) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-16T19:40:47Z

Message from Ruslan Nigmatullin:

Patch Set 13:

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

…pshot Use treap (https://en.wikipedia.org/wiki/Treap) as a persistent map to avoid copying s.goFiles across generations. Maintain an additional s.parseKeysByURIMap to avoid scanning s.goFiles on individual file's content invalidation. This on average reduces didChange latency on internal codebase from 160ms to 150ms. In a followup the same approach can be generically extended to avoid copying s.files and s.packages. Updates golang/go#45686 Change-Id: Ic4a9b3c8fb2b66256f224adf9896ddcaaa6865b1 GitHub-Last-Rev: b211d6c GitHub-Pull-Request: golang#382

gopherbot · 2022-06-17T01:25:41Z

This PR (HEAD: a284a97) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-17T18:47:57Z

Message from Hyang-Ah Hana Kim:

Patch Set 14:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-17T22:17:56Z

Message from Ruslan Nigmatullin:

Patch Set 14:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-21T21:19:09Z

Message from Robert Findley:

Patch Set 14: Run-TryBot+1

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-21T21:23:31Z

Message from Gopher Robot:

Patch Set 14:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-21T21:23:32Z

Message from kokoro:

Patch Set 14:

Kokoro presubmit build starting for golang/tools/gopls-legacy/presubmit
Logs at:
https://source.cloud.google.com/results/invocations/f7b718d5-9141-4c18-9e86-5556cc94950b

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-21T21:23:33Z

Message from kokoro:

Patch Set 14: gopls-CI+1

Kokoro presubmit build finished with status: SUCCESS
Logs at: https://source.cloud.google.com/results/invocations/f7b718d5-9141-4c18-9e86-5556cc94950b

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-21T21:42:01Z

Message from Gopher Robot:

Patch Set 14: TryBot-Result+1

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-21T23:41:12Z

Message from Robert Findley:

Patch Set 14:

(11 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

…pshot Use treap (https://en.wikipedia.org/wiki/Treap) as a persistent map to avoid copying s.goFiles across generations. Maintain an additional s.parseKeysByURIMap to avoid scanning s.goFiles on individual file's content invalidation. This on average reduces didChange latency on internal codebase from 160ms to 150ms. In a followup the same approach can be generically extended to avoid copying s.files and s.packages. Updates golang/go#45686 Change-Id: Ic4a9b3c8fb2b66256f224adf9896ddcaaa6865b1 GitHub-Last-Rev: b211d6c GitHub-Pull-Request: golang#382

Change-Id: If949bf9aac8f60f09d7969feaf6e9f67030400af

gopherbot · 2022-06-22T00:22:47Z

This PR (HEAD: 0abd257) has been imported to Gerrit for code review.

Please visit https://go-review.googlesource.com/c/tools/+/411554 to see it.

Tip: You can toggle comments from me using the comments slash command (e.g. /comments off)
See the Wiki page for more info

gopherbot · 2022-06-22T00:26:41Z

Message from Ruslan Nigmatullin:

Patch Set 16:

(10 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-22T21:03:44Z

Message from Robert Findley:

Patch Set 16: Run-TryBot+1 Code-Review+2

(2 comments)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-22T21:03:45Z

Message from Gopher Robot:

Patch Set 16:

(1 comment)

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-22T21:03:46Z

Message from kokoro:

Patch Set 16:

Kokoro presubmit build starting for golang/tools/gopls-legacy/presubmit
Logs at:
https://source.cloud.google.com/results/invocations/f1285bdc-8ee8-4178-b32a-91b9e255db95

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

gopherbot · 2022-06-22T21:07:59Z

Message from kokoro:

Patch Set 16: gopls-CI+1

Kokoro presubmit build finished with status: SUCCESS
Logs at: https://source.cloud.google.com/results/invocations/f1285bdc-8ee8-4178-b32a-91b9e255db95

Please don’t reply on this GitHub thread. Visit golang.org/cl/411554.
After addressing review feedback, remember to publish your drafts!

…pshot Use treap (https://en.wikipedia.org/wiki/Treap) as a persistent map to avoid copying s.goFiles across generations. Maintain an additional s.parseKeysByURIMap to avoid scanning s.goFiles on individual file's content invalidation. This on average reduces didChange latency on internal codebase from 160ms to 150ms. In a followup the same approach can be used to avoid copying s.files, s.packages, and s.knownSubdirs. Updates golang/go#45686 Change-Id: Ic4a9b3c8fb2b66256f224adf9896ddcaaa6865b1 GitHub-Last-Rev: 0abd257 GitHub-Pull-Request: #382 Reviewed-on: https://go-review.googlesource.com/c/tools/+/411554 Reviewed-by: Robert Findley <rfindley@google.com> gopls-CI: kokoro <noreply+kokoro@google.com> Reviewed-by: Alan Donovan <adonovan@google.com> Run-TryBot: Robert Findley <rfindley@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

gopherbot · 2022-06-22T21:16:41Z

This PR is being closed because golang.org/cl/411554 has been merged.

euroelessar mentioned this pull request Jun 9, 2022

x/tools/gopls: add ability to avoid scanning an entire workspace golang/go#53278

Closed

euroelessar force-pushed the snapshot-optimize-memoize branch from 8c25760 to d6691bc Compare June 10, 2022 18:33

euroelessar changed the title ~~gopls: Use persistent map for storing gofiles in the snapshot~~ internal/lsp/cache: use persistent map for storing gofiles in the snapshot Jun 13, 2022

euroelessar force-pushed the snapshot-optimize-memoize branch from d6691bc to fcf814c Compare June 13, 2022 20:44

euroelessar force-pushed the snapshot-optimize-memoize branch from 9b6e3e1 to f0b31fd Compare June 15, 2022 23:57

euroelessar force-pushed the snapshot-optimize-memoize branch from 0e5720f to b211d6c Compare June 16, 2022 19:29

euroelessar force-pushed the snapshot-optimize-memoize branch from b211d6c to a284a97 Compare June 17, 2022 01:23

Ruslan Nigmatullin added 2 commits June 21, 2022 23:52

cr

0abd257

Change-Id: If949bf9aac8f60f09d7969feaf6e9f67030400af

euroelessar force-pushed the snapshot-optimize-memoize branch from a284a97 to 0abd257 Compare June 22, 2022 00:20

gopherbot closed this Jun 22, 2022

internal/lsp/cache: use persistent map for storing gofiles in the snapshot #382

internal/lsp/cache: use persistent map for storing gofiles in the snapshot #382

Conversation

euroelessar commented Jun 9, 2022 • edited Loading

gopherbot commented Jun 9, 2022

gopherbot commented Jun 9, 2022

euroelessar commented Jun 9, 2022

gopherbot commented Jun 10, 2022

gopherbot commented Jun 10, 2022

gopherbot commented Jun 10, 2022

gopherbot commented Jun 10, 2022

gopherbot commented Jun 10, 2022

gopherbot commented Jun 13, 2022

gopherbot commented Jun 13, 2022

gopherbot commented Jun 13, 2022

gopherbot commented Jun 13, 2022

gopherbot commented Jun 14, 2022

gopherbot commented Jun 14, 2022

gopherbot commented Jun 14, 2022

gopherbot commented Jun 15, 2022

gopherbot commented Jun 16, 2022

gopherbot commented Jun 16, 2022

gopherbot commented Jun 16, 2022

gopherbot commented Jun 16, 2022

gopherbot commented Jun 16, 2022

gopherbot commented Jun 16, 2022

gopherbot commented Jun 17, 2022

gopherbot commented Jun 17, 2022

gopherbot commented Jun 17, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 21, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

gopherbot commented Jun 22, 2022

euroelessar commented Jun 9, 2022 •

edited

Loading