Skip to content

API docs: introduce running tally tables for optimized stats/counting

Warren Gifford requested to merge sg/apidocs-materialized into main

Created by: slimsag

In this PR, we introduce a few running tally tables whose names are self-describing:

  • lsif_data_apidocs_num_pages - used to optimize a query in lsifstore.
  • lsif_data_apidocs_num_dumps - generally useful info, and used by the OOB migration.
  • lsif_data_apidocs_num_search_results_public - generally useful info I would like to expose in the future
  • lsif_data_apidocs_num_search_results_private - generally useful info I would like to expose in the future

We also introduce a running tally table lsif_data_apidocs_num_dumps_indexed which records the number of LSIF dumps that have been indexed for search by the OOB migrator, making the Progress percentage calculation for the API docs' OOB search migrator very cheap - as it had previously caused high CPU usage in codeintel-db due to the expensive count() involved. See #25856 (closed)

Additionally, this concludes the major DB-schema rework I have been doing for the API docs search backend and re-enables API docs search by default, so that indexing will once again continue on Sourcegraph.com.

Fixes #25856 (closed)

Helps #21938

Merge request reports