Skip to content

insights: prune stream insights snapshots

Administrator requested to merge insights/prune-stream-snapshots into main

Created by: leonore

closes #32969 (closed).

We prune the snapshots table when adding new data series to search insights (not capture group insights). A snapshot is the latest retrieved data series for an insight, and an insight is resolved from the union between series_points and series_points_snapshots.

Search stream insights are a backend-enabled only functionality for which we forgot to add this logic to prune the snapshots table.

This also moves some helpers around in work_handler as the file was hard to read in a logical way.

Test plan

The behaviour can be reproduced as follows, keeping in mind that search stream insights can only be enabled by going into the database at the moment.

  1. Create a search insight
  2. Set it to search stream manually
    • To enable streaming for an insight, its capture group method should be set to search-stream and thebackfill_queued_at timestamp should be nulled. More specific instructions
  3. Run a few snapshots by setting the next snapshot after timestamp to NOW() and re-running the query runner.
  4. Observe database and insights state before, and after the change.

Screenshots: Multiple values for the same series, at different times: Step 4 - observe there are duplicate snapshots After fix, values get overwritten: Screenshot 2022-04-07 at 14 49 20 Screenshot 2022-04-07 at 14 48 20

Merge request reports

Loading