insights: prune stream insights snapshots
Created by: leonore
closes #32969 (closed).
We prune the snapshots table when adding new data series to search insights (not capture group insights). A snapshot is the latest retrieved data series for an insight, and an insight is resolved from the union between series_points
and series_points_snapshots
.
Search stream insights are a backend-enabled only functionality for which we forgot to add this logic to prune the snapshots table.
This also moves some helpers around in work_handler
as the file was hard to read in a logical way.
Test plan
The behaviour can be reproduced as follows, keeping in mind that search stream insights can only be enabled by going into the database at the moment.
- Create a search insight
- Set it to search stream manually
- To enable streaming for an insight, its capture group method should be set to
search-stream
and thebackfill_queued_at
timestamp should be nulled. More specific instructions
- To enable streaming for an insight, its capture group method should be set to
- Run a few snapshots by setting the next snapshot after timestamp to
NOW()
and re-running the query runner. - Observe database and insights state before, and after the change.
Screenshots: Multiple values for the same series, at different times: After fix, values get overwritten: