Something went wrong on our end. Please try again.
Created by: efritz
The current LSIF backend uses SQLite as its data store for everything: each repo/commit pair has its own SQLite database (written once by one process and read many times by many processes), as well as a cross-repository database (written by the worker and read many times by many processes).
We will soon have a need for many writers (both the api and worker) to access the cross-repository database concurrently, which will lead to (1) contention on a disks-based database, and (2) higher chance of corruption (journaling modes and exclusive locks still do not solve this problem). This work enables https://github.com/sourcegraph/sourcegraph/pull/5691 to be implemented without chance of data corruption due to multiple writers.
To get ahead of this, this PR splits the data that can be written concurrently by multiple processes to point at Postgres instead of SQLite. In dev, this tablespace can be shared with the same postgres instance used by the frontend/repo-updater/mangement-console. In production, it should likely have its own instance. Tagging distribution (@slimsag) and core service (@tsenart) members that commented on this distribution issue in RFC 25.