insights: retry on search shard-timeout alert
Created by: leonore
as mentioned in https://github.com/sourcegraph/sourcegraph/issues/34613#issuecomment-1129049083 we're getting issues when backfilling insights where we miss data because of timeouts. subsequent hits should however work.
search does propagate a shard-timeout alert that we can use to retry in such cases, we should then hit some caching that should allow us to get the data required.
we can follow test cases mentioned in the issue linked above to confirm this has helped, otherwise will have to explore other avenues.