monitoring: add pg connection usage
Created by: michaellzc
We have no alerting when all connections are used. This PR added a new panel to show percentage of in use connection and added two alerts
- warning: 0.8 (high usage)
- critical: 1.0 (no new connection can be created)
related: https://sourcegraph.slack.com/archives/C02KX975BDG/p1650554603472829 (I scale up the number of worker replica and used up all connections, and other services are struggling to make new connection with postgres)
ideally we will use stuff like pgbouncer in production, but I figure some on-prem customers still use the bundle database and this would be useful
Test plan
sg run redis-postgresOpen a new tab
sg start monitoring Go to http://localhost:3370/-/debug/grafana/d/postgres/postgres?orgId=1
you should see
