search: always limit commit and diff to less than 10,000 repos
Created by: keegancsmith
We have been noticing spikes in search latency on Sourcegraph.com. Turns out the root cause is saved searches doing diff searches. Ops Log. The diff searches would be run over 200k repos, and would constantly fail and keep getting retried.
In https://github.com/sourcegraph/sourcegraph/issues/7215 (https://github.com/sourcegraph/sourcegraph/pull/7272) we removed any limit if before:
or after:
is specified. This was to ensure saved searches worked. However, we need some limit since diff search won't work over 200k repos. This only recently become a problem due to us increasing default repos from 10k to 200k. This PR introduces a limit of 10,000. This may affect customers. If so I can increase to a higher number.
Additionally this PR enforces these limits if we search commits at all. For example type:diff type:path foo
will try search 200k repos with diff search. If you try this out it returns, but that is due to some weird timeout behaviour. We don't report diff didn't work. I think its likely better to just alert to the user and not search.
cc people who have investigated this @pecigonzalo @bobheadxi @daxmc99