Skip to content

insights: queued queries sometimes go from processesing state directly to failed state

Created by: coury-clark

I've observed some insights queries that are queued up jump directly from processing to failed, without attempting to retry. These queries have no failure message other than failed to process which is the standard output from the queue manager when it resets queries at their max resets; however, this query has a reset count of 0 and a max of 3.

Example record:

id series_id search_query state failure_message started_at finished_at process_after num_resets num_failures execution_logs record_time worker_hostname last_heartbeat_at priority cost persist_mode
18360400 s:027C06FB68E7A4F28A89577C066C0299A75A414A0EA04A3F79E3C4901EECA1C3 campaigns file:. count:all failed failed to process 2021-09-09 22:08:11.554816 +00:00 2021-09-09 22:10:48.815050 +00:00 NULL 0 0 NULL NULL worker-b9d8d88c9-gbg4l 2021-09-09 22:09:03.903731 +00:00 10 500 snapshot