monitoring: no easy way to trigger alerts for testing

Created by: bobheadxi

This is a pain point I've felt during dev, but I was able to work around this by changing thresholds manually

A bit trickier for customers - to test the alertmanager side of things (notifier config), they can do something like https://gist.github.com/cherti/61ec48deaaab7d288c9fcf17e700853a at the moment, with url='http://localhost:30090/alertmanager/api/v1/alerts' and kubectl port-forward svc/prometheus 30090, but this is not ideal, and the labels won't match what we actually have

We might want a way to say "create a test critical alert" easily. This could be a dummy 0/1 metric that has a toggle via API or UI or similar