Skip to content

search: always limit commit and diff to less than 10,000 repos

Warren Gifford requested to merge k/after-repo-limit into main

Created by: keegancsmith

We have been noticing spikes in search latency on Sourcegraph.com. Turns out the root cause is saved searches doing diff searches. Ops Log. The diff searches would be run over 200k repos, and would constantly fail and keep getting retried.

In https://github.com/sourcegraph/sourcegraph/issues/7215 (https://github.com/sourcegraph/sourcegraph/pull/7272) we removed any limit if before: or after: is specified. This was to ensure saved searches worked. However, we need some limit since diff search won't work over 200k repos. This only recently become a problem due to us increasing default repos from 10k to 200k. This PR introduces a limit of 10,000. This may affect customers. If so I can increase to a higher number.

Additionally this PR enforces these limits if we search commits at all. For example type:diff type:path foo will try search 200k repos with diff search. If you try this out it returns, but that is due to some weird timeout behaviour. We don't report diff didn't work. I think its likely better to just alert to the user and not search.

cc people who have investigated this @pecigonzalo @bobheadxi @daxmc99

Merge request reports

Loading