If searcher zip archives are corrupted, we should gracefully handle it

Created by: slimsag

It is possible to e.g. shutdown a searcher instance while a zip archive from gitserver is being transferred to it, resulting in empty or corrupt zip files on disk. This leads to search errors like:

1 error occurred: * text search failed: failed to search stash.example.com/myorg/myrepo@: zip: not a valid zip file

The workaround is to find and remove the corrupt zip by running on each searcher instance a command that will identify the corrupt zips and delete them:

find /mnt/cache/searcher/searcher-archives -type f | xargs -I % sh -c 'unzip -lqq % >/dev/null 2>&1 || (echo "removing corrupt archive %" && rm %)'

We should instead automatically do this so our users do not have to. i.e., if a zip archive is invalid we should request it from gitserver again.

cc @keegancsmith