Skip to content

lsif typed syntax highlighter v0.1

Administrator requested to merge tjdevries/lsif_typed_syntax_highlighter_v1 into main

Created by: tjdevries

@olafurpg use this PR instead. The other PR had some mess-ups and some other stuff from trying to update deps (but not happening in that PR anymore because @varungandhi-src is going to do some updates in another PR).

You should be able to just run this locally and then make requests against localhost:8000/lsif and get the data key of the JSON result with the bytes of the lsif proto. There was some problems with getting into JSON at this point due to the version of the protobuf crate we're using (could use an alpha one and get the JSON support, but we can chat about that later).

  • tree-sitter grammar compilation as part of the build
  • snapshot testing infrastructure
  • language injections
  • basic client-side highlighting
  • Separate tree-sitter and syntect namespaces
  • Rust LSIF Typed bindings
  • Get Cargo on buildkite-agent Docker image
  • Get new syntax kinds
  • Update SyntaxKind with new feedback from Olaf, as well as regularizing names
  • Render new syntax kinds in client-side highlighter
    • Add new hl- groups and explain how they get generated
    • hl- groups: Light mode
    • hl- groups: Dark mode
  • Add user setting for tree-sitter highlighting
  • Clean up how we inlined gosyntect into the sourcegraph/sourcegraph repo
    • Current status is acceptable for merge, but will follow up to delete unused options
  • Make sure everything is actually base64 encoded where we say it would be (or don't encode)
  • Update docs for syntax-highlighter README
  • Run e2e tests before merging (push to a branch named main-dry-run/SOMETHING)

Follow up PRs:

  • Observability for tree-sitter requests (or other things?) #30984
  • Highlight arguments: #30995
  • Delete all deprecated arguments for gosyntect / syntect_server #31202

Test Plan

  • Toggle on and off tree sitter feature and check Go files, to make sure they work.
  • Test that other files still work.

Merge request reports