prometheus: support regex silencing of alerts

Created by: bobheadxi

This change allows us to perform regex silences in observability.silenceAlerts.

Pretty much every low-util alert is firing on k8s.sgdev.org: https://k8s.sgdev.org/-/debug/grafana/?viewPanel=11&orgId=1 so this would be useful for silencing those there (I don't think we want to scale down this particular deployment).