insights: Modify historical recorder to execute based on persisted state

Created by: coury-clark

Currently, the historical recorder builds data frames starting from the current time and progressing backward to some pre-defined maximum time. This results in many duplicate queries queued (many millions on k8s).

1. At the minimum historical recorder should only work up to the moment the insight started recording indexed intervals.
2. Maintaining some state about which intervals have already been analyzed would allow this recorder to check if it needs to analyze a given frame
3. The frames to analyze should be built from the time the insight series was created instead of the current time