insights: only query and record values of repositores that have changed
Created by: coury-clark
Currently the code insights backend runs queries and records for all (repository, interval, series) tuples. This causes many unnecessary searches against repositories that have not changed within the interval. The approach is outlined in RFC 392 to compress the queries and recordings and assume that a recorded value at a given time T can be resolved by the most recent timestamp <= T.
-
New tables for commit index + migration script -
Background job to populate index & keep index up to date -
Site setting / feature flag to disable this index for large installations (should fallback to querying every interval)#22036 (closed) -
Integrate with historical backfillter to preemptively remove frames that will not change #22298 -
Update insights query mechanism to return the correct result if a data point is missing for an interval (pick latest) #22298 -
Documentation
Estimated to be completed by: 2021-06-21