observability: Alerting should be easily configured through a file or site configuration

Created by: slimsag

Meta tracking issue to make alerting configuration nicer than a painful series of steps through the Grafana UI; I am pretty sure another exists for this already as we are working on this - I just couldn't find it.