Skip to content

zoekt: Cleanup failed shards

Created by: keegancsmith

On sourcegraph.com we have the OOM killer coming in and killing an index while it is running. A side effect of this is most of the disk space on sourcegraph.com's index-search pod is taken up by half written files. They have a name like:

github.com%2Fzeit%2Fnext.js_v16.00000.zoekt976944826

More generally they match the glob *.zoekt[0-9]*.

A RFC detailing how to tackle this problem should be created. A suggestion I have is using an approach similiar to what we do on gitserver. We use a unique temporary directory per gitserver process. If any other directory matches that directories name on startup, we delete it. This may not be good enough.