insights: discovery: add a way to efficiently iterate over all repos on Sourcegraph

Created by: slimsag

For building historical data for insights, we need a way to efficiently walk over all repositories on Sourcegraph. In the case of customer deployments, this needs to be efficient and done slowly so as to not put any substantial pressure on the DB. In the case of Sourcegraph.com, we obviously do not want to gather historical data for all repositories that exist there. Thus, we use the same subset that are chosen for search indexing (the poorly-named default_repos table/store.)

Helps #18398 (closed)

Signed-off-by: Stephen Gutekanst [email protected]