Skip to content

insights: refactor just in time converter to account for correct series backfill starts

Warren Gifford requested to merge cw/refresh_scoped_insights into main

Created by: chwarwick

Adds a new field to the series to track if the series either needs to be converted from just in time or needs to be re-backfilled.

Adjusts the conversion logic to replace the seriesID so that re-backfilled series won't have extra conflicting points.

Test plan

Manually tested the following 3 scenarios:

  1. A db with a mix of all repo insights that haven't yet backfilled and just in time insights with a created date 4 months ago
  2. A db with a mix of all repo insights and scoped insights with a created date 4 months ago that previously backfilled
  3. A db with all repo insights and a mix of scoped insights that previously backfilled and just in time insights all of which have a created date of 4 months ago

In each scenario results passed as expected:

  • The scoped insights properly moved in the processing state and the filled with data
  • The most recent data point for the scoped insights corresponded to the current date as expected
  • The db flag for migrating was properly updated and the scoped insights did not refresh again on future runs as expected
  • Verified the count for a point on the series

In addition tested 2 additional paths:

  1. Manually triggering multiple restarts during the conversion process
  2. Forcing 1 of the insights to fail the backfill step
  • This resulted in all the other insights properly completing and the errored insight stopping after 10 attempts as expected.

Merge request reports