insights: flag query as dirty if it is terminally 'failed'
- Truncate descriptions
Created by: coury-clark
To generate an insight data series, search queries are queued for (repository, time) tuples. The queries are processed asynchronously in the background from a table that serves as the queue. The states that the records can move in is as follows:
queued -> processing -> completed
queued -> processing -> errored
queued -> processing -> failed
errored -> processing -> errored
errored -> processing -> failed
There are two paths that result in a terminal state of failed
which has the semantic that the record has exceeded the retry policy and is considered unprocessable.
It will be useful to inform users that their insight has potentially misleading values. A table exists called insight_dirty_queries
that is used to record queries that were not wholly successful. We should mark any queries that enter the failed
state as dirty and expose reason
metadata describing as much of the problem as possible.

