monitoring: some observables end up with multiple metrics
Created by: bobheadxi
Confirmed in Cloud as well:
level=warn ts=2021-01-25T01:45:55.455Z caller=manager.go:598 component="rule manager" group=zoekt-webserver msg="Evaluating rule failed" rule="alert: warning_zoekt-webserver_provisioning_container_memory_usage_short_term\nexpr: (max_over_time(cadvisor_container_memory_usage_percentage_total{name=~\"^zoekt-webserver.*\"}[5m])) >= 90\nlabels:\n alert_type: builtin\n description: 'zoekt-webserver: 90%+ container memory usage (5m maximum) by instance'\n grafana_panel_id: \"100211\"\n level: warning\n name: provisioning_container_memory_usage_short_term\n owner: search\n service_name: zoekt-webserver\n" err="vector contains metrics with the same labelset after applying alert labels"
This seems related to https://github.com/sourcegraph/sourcegraph/pull/11985 and https://github.com/sourcegraph/sourcegraph/issues/11571#issuecomment-654571953, functionality that was removed in 3.24 (https://github.com/sourcegraph/sourcegraph/pull/17087). We might need that min/max after all