search: significantly improve the performance of file search suggestions

Created by: slimsag

This change significantly improves the performance and code quality of the implementation providing file search suggestions.

It reduces IO load on gitserver, but primarily removes load from repo-updater and redis which occurred due to a bug in the old implementation trying to update all repositories involved in the search.

Fixes #2089 (closed)

How?

Most importantly, the old implementation issued requests erroneously which caused repo-updater, redis, and gitserver more IO load than was optimal. You can read more about this in the linked issue above.

File search suggestions now also benefit from the same zoekt indexing that we do for search otherwise, because we are now using the same code path that provides actual search results.

In sourcegraph/server instances, which do not use indexed search by default, performance is roughly identical to before.

Code quality gains

Prior to this change file search suggestions were provided by an entirely separate code path. This old code path was responsible for listing all files in every repository your query matched, then matching those files against your search query. In contrast, after this change:

1. File search suggestions are provided by the same code-path which provides text search results.
2. Because the code-path is the same, you no longer get different ordering in search suggestions compared to when you actually run the query.

Change in fuzzy matching behavior / more consistent matching

Before this change, you could type src/http.go (internally converted to file:src/http.go) and get fuzzy suggestions matching src/net/http/http.go even though your query was missing the net/http part of the path.

After this change, you must type a substring match, or use regex. i.e. src/http.go will now not fuzzy match / provide any suggestions, but these would:

• http.go
• http/http.go
• net/http/http.go
• src/net/http/http.go
• src.*http.go

The prior behavior was better in some regards because it acted more like your editor quick file open panel. However, the inconsistency with the results you would get when running your query was also bad. For example:

1. Visit https://sourcegraph.com/github.com/sourcegraph/sourcegraph@v3.0.0
2. Type file:dev/main.go in the search input and you will see two suggestions.
3. Actually press enter and run the query, you will recieve no results!

We now have consistency and a shared implementation. In the future, we could gain this behavior back by changing the file: match operator to do fuzzy editor-like matching, instead of (or in addition to) using regex. However, it is unclear to me if doing that would be worth it or if this behavior is generally good enough (I suspect it is).