Skip to content

codeintel: Refactor SQLite layer, introduce migration mechanism

Administrator requested to merge bundle-sqlite-migration-base into master

Created by: efritz

This work is necessary to change our SQLite db schema for:

  • tracking additional information (e.g. code insights)
  • applying better encodings without breaking all existing uploads
  • changing datastructures within the db for efficiency

This PR adds migrations, but does not do so in a way that affects the current schemas (we only add a version table).

Please review by commit:

  • (1) Add a bundle metadata type instead of passing around raw int/string values in Read/WriteMeta functions. We also drop the lsif/sourcegraph versions here as they're not read anywhere at this time and keep only the numResultChunks value. We may use this type to easily include additional metadata in the future.

  • (2) Introduce a *Store struct that backs the SQLite reader and writer. This really just reflects transactions and scanning in the same way as the codeintel/db layer for consistency.

  • (3) Remove the writer Flush method and instead have each Write method flush its own inserter. This cleans up the writer interface a bit.

  • (4) Add migrations. These are applied in the reader when a database is first opened. The mechanism here is hand-rolled instead of using a third-party tool mainly because the migrations we will be performing in the future are not possible to do only in SQL: we'll want to read rows from the database to re-encode them, which requires us to be able to run arbitrary Go code. We also do not care about migrating databases backwards, so the code introduced here is fairly simple.

The remaining commits are test coverage and cleanup commits.

Note that migration does have test coverage due to it being run in the reader from an old db file, but doesn't have any explicit tests.

Merge request reports