insights: Prioritize queries so that more than a single series can populate at once

Created by: coury-clark

Currently the insights backend queues all work without priority. Insight series that have a large number of queries to execute, such as a historical data frame can require many thousands of searches to generate a single data point. Other insight data series will have to wait behind all of those queries to finish executing before they can populate at all.

Data frames closer to the present time should be prioritized higher so that we can quickly deliver value to an insights user.

Proposal

Insight queries could be prioritized with a numeric value where 0 represents the highest priority and larger values represent lower priority.

Historical queries will prioritize using the number of days prior to the creation date Current queries will prioritize with 0 (or perhaps 1 to leave room for higher priority?)

This will cause all insight series to grow to the same width at the same time, and evenly distribute queries across each series.