search: Add possibilty to use codemirror as search query input

Created by: fkling

This PR explores the possibility to use CodeMirror (CM) for our query input. I'm creating this PR to share and explain this implementation but I don't assume that it can be merged as is.

The expectation is that we can customize CM better to fulfill our needs. That will likely require is to implement a lot of the behavior we currently get for free from Monaco, but I think it will be worth it. Currently we have to jump through hoops to disable some of the undesirable behavior of Monaco (example, example).

Rollout: With this PR I'm introducing the editor experimental feature flag which accepts pre-defined string values for choosing which editor to use. LazyMonacoQueryInput uses that value to determine which editor to load. This can also be used in other places where we want to switch to Monaco in the future. I updated all places that use LazyMonacoQueryInput to the best of my knowledge. My plan would be to get some people from SG to try this, get early feedback and then open it to more people or even the whole company.

About this prototype

My current plan is to replicate the existing behavior as closely as possible while tweaking some smaller aspects to demonstrate what we will be able to do with CM, so that it would be possible to use it as a drop in replacement (via a feature flag).

Since this is a proof of concept, I simply copied some of the existing logic we have for Monaco where it was necessary. To make this more production ready I'd refactor the existing code where it makes sense.

CM provides a "proper" way to support languages but I didn't want to spend the time yet to figure out how to integrate our own query parser with that, so I do syntax highlighting "manually".

The query input consists of three major parts:

• A hook (useCodemirror) responsible for creating an editor instance and keeping it up-to-date.
• A "core" component (CodemirrorQueryInput) which tries to provide a relative generic interface. It handles core features such as query syntax highlighting.
• A "facade" component (CodemirrorMonacoFacade) which accepts the same props as MonacoQueryInput and creates the necessary extensions. This component would probably be refactored at some point when we don't have to support Monaco anymore to provide a better API.

Here is a video demonstrating the current state. It shows two new things: "Highlighting" the currently focused filter with a simple background color change and rendering diagnostic messages with Markdown.

https://user-images.githubusercontent.com/179026/157886020-7f8233dd-8954-4898-a793-24d83d5fb5e4.mp4

Quick intro to CM architecture

I highly recommend reading CM's own documentation about its architecture.

CM maintains an internal editor state that gets updated via transactions.

Almost all behavior is implemented via extensions, and there are different kinds of them. Extensions have the ability to extend the editor state via state fields. Other extensions, e.g. view plugins can access information in these fields and make changes to the UI.

Then there are facets which are basically values derived from fields and other facets. As an example, this implementation stores the parsed search query in a facet (it updates whenever the editor content changes) and other extensions, such as the one for syntax highlighting, generate new decorations whenever that facet changes.

TODO

(some of those might be addresses in later PRs, will create issues as necessary)

• Add syntax highlighting
• Add support for query diagnostics
• Add support for suggestions
• Icons for suggestions
• Suggestions order
• Suggestions filter (doesn't show suggestions for e.g. repo:^. Monaco has an extra filterText field for this, but CM doesn't)
• Accept suggestion on Tab instead of enter
• Add proper support for dark theme
• Handle KEYBOARD_SHORTCUT_FOCUS_SEARCHBAR
• Handle onHandleFuzzyFinder
• Hide vertical scroll bar
• Replace line breaks with spaces (?) (this would be useful if someone pasted a multiline query into a single-line input)

I think that despite not everything working exactly as before, we could already merge this to get some early feedback.

Test Plan

Just manual testing for now. The feature is gated behind a feature flag. Depending on how Sourcegraph is run locally it might be necessary to override the code in LazyMonacoQueryInput to always load Codemirror (e.g. editorComponent === 'codmirror6' => true).

Some things I tested:

• Added disabled={true} to one of the fields in MonacoField.story.tsx to test read-only view.
• Entered multiline queries in the MonacField stories and the query-based search context input via Enter.
• Verified that Enter submits the query in the main query input and that Shift+Enter doesn't insert new lines.