searcher: not respecting on disk cache limit

Created by: keegancsmith

There are instances of searcher being evicted at customers by kubernetes. We believe the root cause is searcher not respecting the cache limits. The purpose of this issue is to track the root causing of this, provide work arounds and ultimately resolve this issue.

Some related issues:

• Gracefully handle limited shared cache space #627
• searcher and symbols deployments should switch to statefulsets #32279
• Unindexed search archive cache is not monitored #9796
• Searcher pod getting evicted https://github.com/sourcegraph/customer/issues/740