search: add value scanner
Created by: rvantonder
Stacked on #9775.
This PR makes the scanner/parser feature complete compared to our existing query scanner. The notable change is that it allows to interpret delimited/quoted values for parameters and patterns, like repo:"foo bar baz"
, and builds in escape sequence recognition for certain special syntaxes like :
and \n
.
This implementation improves on our current logic, and gracefully handles nasty issues like https://github.com/sourcegraph/sourcegraph/issues/9147 because it will also scan/parse heuristically.
There are many more improvements and nice things we can do to improve the implementation and options (filed #9784), but this is sufficient for now.