monitoring: Alert on invalid indexes

Created by: efritz

There are several places where we've used CREATE INDEX CONCURRENTLY in our migrations. Developers may not know the caveats, and if they did we have not yet addressed them.

Indexes created concurrently avoid a full table lock (so that reads and writes can still proceed). We've seem to have fallen into a siren's call by leaning on this feature, as concurrently created indexes can fail just ... in the background without indicating that has done so. Describing the table will show an invalid index, but unless you're looking for it, the query planner will just avoid the index on selects and that's the biggest hint you'll get that something is wrong.

This adds a new alarm for all indexes that are (a) invalid, and (b) are associated with an in-progress index creation. This avoid alarms from happening when the index is invalid but still in-progress. This is necessary in Cloud, but even more so on-prem. It is likely that some performance issues have (or will) come from having a broken index on some critical query path.