observability: src_graphql_field_seconds histogram sometimes produces NaN values
Created by: slimsag
histogram_quantile(0.90, rate(src_graphql_field_seconds_bucket{type="Search",field="results",error="false"}[5m]))
and the even more simple:
histogram_quantile(0.95, sum(rate(src_graphql_field_seconds_bucket[5m])) by (le))
returns NaN values sometimes. It's not entirely clear to me why yet, this could be a real bug. I can reproduce it on k8s.sgdev.org but not sourcegraph.com or a dev instance with no data for that metric present.
For now, allow that metric to have NaN values without firing the alert (until the real cause is known).