Skip to content

insights: store state for data series that have had a full backfill iteration

Created by: coury-clark

As a quick solution to historical data series endlessly backfilling, I propose we store a simple flag on each data series that represents when a series has seen a full repo iteration. backfill_queued_at TIMESTAMP

We will modify the backfiller algorithm as such:

  1. Select only data series where backfill_queued_at is null
  2. Once all repositories have been iterated, set backfill_queued_at to current_timestamp for all selected series

This doesn't necessarily mean everything was successful. Ideally, it means everything was queued that should have been, but that doesn't necessarily have to be true either. We can enhance this more in the future to only update series that had zero errors.

Originally posted by @coury-clark in https://github.com/sourcegraph/sourcegraph/issues/22324#issuecomment-890227338