insights: queued queries sometimes go from processesing state directly to failed state
Created by: coury-clark
I've observed some insights queries that are queued up jump directly from processing to failed, without attempting to retry. These queries have no failure message other than failed to process which is the standard output from the queue manager when it resets queries at their max resets; however, this query has a reset count of 0 and a max of 3.
Example record:
| id | series_id | search_query | state | failure_message | started_at | finished_at | process_after | num_resets | num_failures | execution_logs | record_time | worker_hostname | last_heartbeat_at | priority | cost | persist_mode |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 18360400 | s:027C06FB68E7A4F28A89577C066C0299A75A414A0EA04A3F79E3C4901EECA1C3 | campaigns file:. count:all | failed | failed to process | 2021-09-09 22:08:11.554816 +00:00 | 2021-09-09 22:10:48.815050 +00:00 | NULL | 0 | 0 | NULL | NULL | worker-b9d8d88c9-gbg4l | 2021-09-09 22:09:03.903731 +00:00 | 10 | 500 | snapshot |