retry when we have a corrupt archive

Created by: ijsnow

This PR handles corrupted searcher archives by removing the bad zip and retrying. It's unclear why archives would ever be corrupt in the first place so this is only a partial solution. Closes https://github.com/sourcegraph/sourcegraph/issues/1759.

Test plan: Added a test for retry logic.

To manually test:

1. Run a local sourcegraph with dev/launch.sh and add some repositories
2. Ensure there are searcher archives by running a search like index:no test.
3. Run go run ./dev/corrupt-archives/main.go /tmp/searcher-archives to corrupt the archives
4. Execute the search again. You should see search results appear as expected