monitoring: generate alerts as native prometheus alerts, let alert_count depend on alert rules
Created by: bobheadxi
related:
- out-of-band alerts (defined as prometheus alert rules) don't line up with the rest of our alerts in dashboards, etc: https://sourcegraph.slack.com/archives/CJX299FGE/p1595254363107300
- difficult to express certain alert types purely with recording rules (ie
alert_count
): https://github.com/sourcegraph/sourcegraph/issues/12158
potential solution: https://sourcegraph.slack.com/archives/CJX299FGE/p1594994195043000?thread_ts=1594971987.034300&cid=CJX299FGE
we could continue to support alert_count by inverting the dependency - have the alert rule make the query, make alert_count recording rules something like
ALERTS{alertname="blah"} > 0
cc @keegancsmith