Skip to content

insights: Add a background process to calculate backfill_completed_at

Created by: CristinaBirkel

RFC: https://docs.google.com/document/d/1pCoMsWWinaigBktIBeI4g5hnXU2LDivpzLqQ_gjtbeg/edit#

This issue is to do the first step of the work in the RFC.

  1. Add a new column to the schema for the insight_series table for backfill_completed_at.
  2. Create a new background process that runs once ever few hours and queries the insight_query_runner_jobs table to see if any series have finished backfilling that aren't already marked as complete.
  3. If so, take the last completed job and use the timestamp on that to mark the backfill_completed_at timestamp on the series.