Optimize pipeline for LSP request scheduling

[testforstephen]

From the performance analysis, we found that the scheduling conflicts between LSP requests, especially lock conflicts between different WorkspaceJobs, can have a criticial impact on the responsiveness of language server. We definitely need to revisit the use of all WorkspaceJobs in jdtls and remove or shrink scheduling rules of lock.

In jdtls, the following operations would create workspace jobs or scheduling rules.

• ResourcesPlugin.getWorkspace().run(...), this will run in current thread. It has risk to block the lsp4j jsonrpc processor thread, causing all subsequent lsp requests to queue.
• new WorkspaceJob(...), this will run in a seperate thread.
• IResource.refreshLocal(...)

Here are all potential workspace jobs usages.

• BaseDocumentLifeCycleHandler.java
  Known perf issues:
  • Validation/DiagnosticJob --block--> didChange (which runs in lsp4j IO thread) --block--> completion
• WorkspaceEventsHandler.java
• FileEventHandler.java
• ClasspathUpdateHandler.java
• InitHandler.java
• InvisibleProjectImporter.java
• MavenProjectImporter.java
• ProjectsManager.java
• StandardProjectsManager.java
• UpdateClasspathJob.java
• DiagnosticsCommand.java

[testforstephen]

The jsonrpc processor can be blocked by some handlers as below, which can affect the performance and responsiveness of the language server. To reproduce the blocking, open a medium-sized project like https://github.com/google/guava, trigger a full build and then type code for completion while the build is in progress. The completion may sometimes not respond because the jsonrpc processor is stuck on some handlers.

• BaseDocumentLifeCycleHandler
  https://github.com/eclipse/eclipse.jdt.ls/blob/702d59eb54f6bdd5e8bf1efbec3db2856b3a6339/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/handlers/BaseDocumentLifeCycleHandler.java#L339-L348

https://github.com/eclipse/eclipse.jdt.ls/blob/702d59eb54f6bdd5e8bf1efbec3db2856b3a6339/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/handlers/BaseDocumentLifeCycleHandler.java#L354-L374

https://github.com/eclipse/eclipse.jdt.ls/blob/702d59eb54f6bdd5e8bf1efbec3db2856b3a6339/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/handlers/BaseDocumentLifeCycleHandler.java#L324-L333

https://github.com/eclipse/eclipse.jdt.ls/blob/702d59eb54f6bdd5e8bf1efbec3db2856b3a6339/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/handlers/BaseDocumentLifeCycleHandler.java#L309-L319

• WorkspaceEventsHandler
  https://github.com/eclipse/eclipse.jdt.ls/blob/702d59eb54f6bdd5e8bf1efbec3db2856b3a6339/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/handlers/WorkspaceEventsHandler.java#L72

[testforstephen]

When WorkspaceEventsHandler.didChangeWatchedFiles triggers Resource.refreshLocal, this can block the pipeline if the scheduling rule is acquired by other long running job.

"pool-1-thread-1" #51 prio=5 os_prio=0 cpu=9140.62ms elapsed=407.38s tid=0x000001e2ffb20fe0 nid=0x1e10 in Object.wait()  [0x000000a5a54fd000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(java.base@17.0.2/Native Method)
        - waiting on <no object reference available>
        at java.lang.Object.wait(java.base@17.0.2/Object.java:338)
        at org.eclipse.core.internal.jobs.ThreadJob.waitForRun(ThreadJob.java:322)
        - locked <0x0000000633eb8c00> (a java.lang.Object)
        at org.eclipse.core.internal.jobs.ThreadJob.joinRun(ThreadJob.java:208)
        at org.eclipse.core.internal.jobs.ImplicitJobs.begin(ImplicitJobs.java:95)
        at org.eclipse.core.internal.jobs.JobManager.beginRule(JobManager.java:316)
        at org.eclipse.core.internal.resources.WorkManager.checkIn(WorkManager.java:124)
        at org.eclipse.core.internal.resources.Workspace.prepareOperation(Workspace.java:2332)
        at org.eclipse.core.internal.resources.Resource.refreshLocal(Resource.java:1565)
        at org.eclipse.jdt.ls.core.internal.JDTUtils.getFileOrFolder(JDTUtils.java:1210)
        at org.eclipse.jdt.ls.core.internal.managers.StandardProjectsManager.fileChanged(StandardProjectsManager.java:223)
        at org.eclipse.jdt.ls.core.internal.handlers.WorkspaceEventsHandler.didChangeWatchedFiles(WorkspaceEventsHandler.java:115)
        at org.eclipse.jdt.ls.core.internal.handlers.JDTLanguageServer.didChangeWatchedFiles(JDTLanguageServer.java:577)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@17.0.2/Native Method)
        at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@17.0.2/NativeMethodAccessorImpl.java:77)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@17.0.2/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@17.0.2/Method.java:568)
        at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.lambda$null$0(GenericEndpoint.java:65)
        at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint$$Lambda$281/0x000000080109e7a8.apply(Unknown Source)
        at org.eclipse.lsp4j.jsonrpc.services.GenericEndpoint.notify(GenericEndpoint.java:152)
        at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.handleNotification(RemoteEndpoint.java:220)
        at org.eclipse.lsp4j.jsonrpc.RemoteEndpoint.consume(RemoteEndpoint.java:187)
        at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.handleMessage(StreamMessageProducer.java:194)
        at org.eclipse.lsp4j.jsonrpc.json.StreamMessageProducer.listen(StreamMessageProducer.java:94)
        at org.eclipse.lsp4j.jsonrpc.json.ConcurrentMessageProcessor.run(ConcurrentMessageProcessor.java:113)
        at java.util.concurrent.Executors$RunnableAdapter.call(java.base@17.0.2/Executors.java:539)
        at java.util.concurrent.FutureTask.run(java.base@17.0.2/FutureTask.java:264)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.2/ThreadPoolExecutor.java:1136)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.2/ThreadPoolExecutor.java:635)
        at java.lang.Thread.run(java.base@17.0.2/Thread.java:833)

[testforstephen]

One of the optimization challenges is calling resource.refreshLocal(…), which needs a scheduling rule to lock the resource. JDT.LS relies on many patches that call refreshLocal to sync the resource tree with the local file system. This can cause blocking when it runs in the main jsonrpc thread, such as in didClose and didSave. We need to find a way to avoid the blocking of refreshLocal, for example by interrupting a build job and giving refreshLocal higher priority.

https://github.com/eclipse/eclipse.jdt.ls/blob/daa965dd52071ea8ca2142be1bebb4c3672dcc4c/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/handlers/BaseDocumentLifeCycleHandler.java#L505

https://github.com/eclipse/eclipse.jdt.ls/blob/daa965dd52071ea8ca2142be1bebb4c3672dcc4c/org.eclipse.jdt.ls.core/src/org/eclipse/jdt/ls/core/internal/handlers/BaseDocumentLifeCycleHandler.java#L488