Skip to content

codeintel: Change representation of moniker lookups in bundle files

Administrator requested to merge bundle-sqlite-migration into master

Created by: efritz

This PR converts the definitions and references table of SQLite bundles from normalized SQL tables into denormalized json-encoded values keyed by the original primary key of the table. This reduces the number of columns by grouping all (scheme, identifier) values into a single row.

As an added benefit, we now treat SQLite as a simple key-value store on disk, each table having an encoded data field that can be easily swapped for another serialization method.

The converted payloads from the integration tests in internal/cmd/precise-code-intel-test have the following sizes (left=pre, post=right) and show a savings of 35.8% on disk.

   2.4M       1.9M
   2.5M       2.0M
   3.6M       2.4M
   3.6M       2.4M
   3.6M       2.5M
  37.0M      27.0M
  37.0M      28.0M
  37.0M      28.0M
  97.0M      58.0M
  97.0M      58.0M
+ 97.0M      58.0M
-------------------
  417.7M    268.2M

The largest current database file on dotcom is 378.5 MB, and after running the migration on it, it occupies only 54.7 MB.

This change does not significantly decrease insertion or query efficiency.

Merge request reports