search frontend: highlight regex-delimited patterns
Created by: rvantonder
Stacked on https://github.com/sourcegraph/sourcegraph/pull/37912.
Now that the tokens carry extra state, it's easy to case out when the syntax allows a regex pattern like this:
Along with this change I'm making the decision to classify the slashes in /.../ to be the same as parens in regex capture groups (...) since for the most part they "mean" the same from a search perspective. This makes it easier to just reuse existing Token definition and hovers with a slight rephrase. I'm treating this as soft intermediate step, and will follow up with separating (...) and /.../ as a lower-priority issue.
Test plan
App preview:
Check out the client app preview documentation to learn more.