Skip to content

insights: Re-use series points in oob settings migration

Administrator requested to merge insights/api/migrate-replace-ids into main

Created by: CristinaBirkel

Closes https://github.com/sourcegraph/sourcegraph/issues/28505

Description

This PR takes another pass at the logic to re-use series and series points in the oob migration. There are considerations:

  • If a series matches an existing series (which could happen if two insights use the same series,) that matched series gets used instead of a newly created one.
  • If a new backend series is created, we do a find-and-replace on the series_points.series_id so that those series points can get used by this new series (with its new id.) backfill_queued_at also needs to be stamped so that they don't get re-processed.

I decided not to touch the insight_query_runner_jobs because I don't think there's much of a need here. The only edge-case would be if a series is currently processing when we run the oob migration. And the result would be that the insight won't display as "still processing," even while it finishes processing. I don't think this will have a big impact on users and worst case it will be a one-time fluke, but let me know if you disagree.

Testing Steps

I tested this by:

  1. Wiped out my db with:
TRUNCATE insight_view CASCADE;
TRUNCATE insight_series CASCADE;
TRUNCATE dashboard CASCADE;
TRUNCATE series_points CASCADE;
  1. I ran the 10-minute sync job once so that BE insights would populate with the old series_id format
  2. Then I wiped the db again with just:
TRUNCATE insight_view CASCADE;
TRUNCATE insight_series CASCADE;
TRUNCATE dashboard CASCADE;
  1. This leaves us in a state which should be the same as a pre-migration state on an instance.
  2. Then I ran the migration and checked the following:
  • insight_series were re-used when they matched
  • series_points had their ids swapped with the new ones
  • backfill_queued_at was stamped for the BE series
  • Via the UI, the insights loaded up and were not showing as still processing, and looked good

I think this was a pretty good test and the next step will be to see how it does on k8s.

Merge request reports