Performance improvements for workspace resolution

Created by: eseliger

Workspace resolution is currently resource-hungry and can be slow for larger instances.

There are a few things I've identified that could be good improvements here:

• Can we change the .batchignore algorithm to always look at the default branch instead of the target branch?
  • If so, we can add a background worker that runs over all repos changed since the last run to persit the ignored status to a database table for fast lookups
• We can run the search from the worker itself. The search API has been refactored to run on any service now. This will help with resource contention on the frontend while the workspace resolver is running.
  • There are currently two things that need to be solved before we can make it happen:
    • The logic to get user settings is still in the GQL layer, it would need to be factored out (good refactor in general)
    • The worker pod doesn't have all the env vars that are required to run all kinds of searches. Ideally it could get these config values from the frontend instead of us duplicating all the env vars across services.
• Parallelize resolving on sections
• Use a batch method like BatchLog for GetDefaultBranch
• Investigate if we can use gitserver directly instead of search for FindDirectoriesInRepos, this runs for each repository. Ideally this also uses a batch method.