docker-compose distro services can be scaled horizontally

Created by: michaellzc

Feature request description

All services in docker-compose can be scaled horizontally with minimal effort (or they're documented)

Is your feature request related to a problem? If so, please describe.

the root cause of why code-singiths is spamming searcher is out of scope, https://github.com/sourcegraph/sourcegraph/issues/34659

on demo, code-insights query runner is spamming searcher really hard, and we're hitting the limit of a single searcher instance. Vertical scaling no longer helps, and we need to spawn multiple instances to distribute the load, such that every instance will have some breathing room to release the memory.

searcher has to load repo into memory to perform text-based searches, and we're seeing the single searcher instance is struggling.

we may see more services heavily rely on text-based search over indexed search, so being able to scale searcher (or other services) could be really important

Describe alternatives you've considered.

We've tried almost 4x the resources for the searcher container (in fact, it has more resources compared to a single searcher pod on dotcom) but don't see much improvement, we still constantly get high CPU and RAM alerts.

Additional context

learn more slack thread

learn more about why we don't support this right now