Skip to content

codeintel: Keep api doc search current asynchronously

Warren Gifford requested to merge ef/fast-lsif-data-upsert into main

Created by: efritz

Changes:

  • Add a new table that keeps track of the most current dump_id for a (repo, root, language)
  • Remove deletions from the data insertion hot-path. We don't want lock contention clearing old records, so we only append to the table when processing an LSIF record. We now expect that the query path uses the condition: dump_id = (SELECT dump_id FROM lsif_data_doc_search_public WHERE repo_id=outer.repo_id AND dump_root=outer.dump_root AND lang_name_id=outer.lang_name_id) to remove non-relevant records that may exist between cleanup runs.
  • Create a background process that periodically removes the records that would be filtered out by the condition above.

I hope that this restores the PCI worker throughput on Cloud.

Helps #21938

Merge request reports