Language Server Syntax Highlights

[thecoolwinter]

Description

Adds semantic token syntax highlighting to the code file view.

When an LSP is installed and configured for a language type, and has semantic highlights support, CodeEdit will install a new highlight provider on the source editor and begin processing syntax tokens for the file.

Token processing happens asynchronously, and does not replace tree-sitter highlights. This builds off recent work in the source editor to support a hierarchy of highlight providers. Language server highlights are slow but more accurate, so we process them slowly and apply them when they become available.

Details

• Adds a new generic 'language server document' protocol that includes only what the language server code needs to know about a code document. This should solve the coupling issue we had with CodeFileDocument and the language server code. In the future, if we replace CodeFileDocument, it'll be a matter of conforming the new type to the protocol for it to work with the lsp code.
• Reorganizes slightly to group lsp features into their own "Features" folder.
• Adds a new SemanticTokenHighlightProvider type
  • Conforms to the HighlightProviding protocol.
  • Manages receiving edit notifications from the editor and forwards them to the language server service.
• Adds a SemanticTokenMap type
  • Maps LSP semantic token data to a format CodeEdit can read.
  • Reads a LSP's capabilities to determine how to decode that data.
• Adds SemanticTokenStorage
  • Provides an API for applying token deltas, and entire file token data.
  • Manages decoding, re-decoding (when dealing with deltas) and storing semantic tokens.
  • Provides an API for finding semantic tokens quickly.

Related Issues

• closes ✨ Semantic Highlights #1950

Checklist

• I read and understood the contributing guide as well as the code of conduct
• The issues this PR addresses are related to each other
• My changes generate no new warnings
• My code builds and runs on my machine
• My changes are all related to the related issue above
• I documented my code

Screenshots

Live editing demo, note the highlights on the variable types and switch cases.

CodeEdit.mp4

[thecoolwinter]

This is officially ready for review with passing tests!

[tom-ludwig]

This looks really nice! I do have some questions—some are just out of personal interest

[FastestMolasses]

Looks good, will approve after the comments are resolved

[thecoolwinter]

Failing test is due to a bug in CETV that's fixed with this PR. This should be good to merge and would be great to get done.

[austincondiff]

This has been open for a while and seeing as @FastestMolasses's only concern has been fixed, it shouldn't be a problem to merge. Lets move forward. ⏩ I did give it a quick look and it looks good from what I can tell. 👍