Skip to content

observability: relax search latency alerts to match real world

Administrator requested to merge sg/relaxed-alerts into master

Created by: slimsag

This PR relaxes the search P99/P90 latency alerts to more accurately reflect well-behaving instances of Sourcegraph.

Based on 30d average and max value on three real instances:

sourcegraph.com: avg(P99=30s P90=4s) max(P99=30s P90=5s) k8s.sgdev.org: avg(P99=11s P90=7s) max(P99=35s P90=30s) customer: avg(P99=15s P90=10s) max(P99=35s P90=35s)

Max values are assumed to mostly be attributable to heavy API requests, avg appears to be more generally correct browser user experience. sourcegraph.com also assumed to be an outlier, so not used.

Thus, the alerts we configure here are relaxed to:

  • Browser: P99>20s P90>15s
  • API: P99>50s P90>40s

Fixes #9894

Merge request reports