Add global timeouts to Postgres connections
Created by: efritz
Feature request description
Globally add a (sufficiently large) hard timeout on:
- Postgres transactions
- Postgres statements
- Postgres queries
If we don't have any timeouts on dial attempts, we should also fix that (but I believe we do).
Is your feature request related to a problem? If so, please describe.
There were some recent customer issues in which long-running queries were affecting the instance from working correctly. If we were to cap all queries with some sufficiently high default (that isn't crossed under normal operation), it would help the instance recover and may also help identify the queries that are taking too long (e.g., if a live-locked query is found only under some circumstances, this will help detect it).
Describe alternatives you've considered.
Do nothing and continue to have issues fining root causes of database query errors.