migrations: Poll for advisory locks in the runner
Created by: efritz
Pulled from #29831. In this PR, we remove the Lock
method from the flow of the migration runner and rely only on the TryLock
method. Where Lock
was used before, we replace with a polling loop that tries to acquire the lock.
This change will prevent a class of errors in which concurrent index creation live/dead-locks with the migration advisory locks (depending on the number of concurrent migrators, the previous code live-locks AND dead-locks). By refreshing the transaction ID on every attempt of the lock, we ensure that we don't starve out concurrent index creation by holding open long transactions while waiting for migrations to finish.