codeintel: alert when all executor jobs are failing

Review changes
Download
Patches
Plain diff

Merged codeintel: alert when all executor jobs are failing

nsc-ef/executor-errors-over-time into main

Overview 16
Commits 5
Pipelines 0
Changes 8

Merged Administrator requested to merge nsc-ef/executor-errors-over-time into main 3 years ago

Overview 16
Commits 5
Pipelines 0
Changes 8

Created by: Strum355

Creates alert for executors error rate that alerts when the rate of errors is 100%, indicating some global misconfiguration (as happened before with src-cli related issues).

The alert is a bit special in that it uses a different query to the panel, one based on last_over_time aggregate. We do this as we dont want the alert to mark itself as resolved if there happens to be a period over the defined window where there are no auto indexing jobs (aka when the error rate is "technically" < 100%).

The screenshot below illustrates how the alert query maintains the last value over a predefined window, so that if no executor jobs are processing but over the error rate was 100% before, we will continue alerting as the absence of running jobs does not imply the issue is resolved.

Closes https://github.com/sourcegraph/sourcegraph/issues/30494

Test plan

Only modifies dashboards/alerts, n/a

Merge request reports

Activity

Filter activity

Approvals
Assignees & reviewers
Comments (from bots)
Comments (from users)
Commits & branches
Edits
Labels
Lock status
Mentions
Merge request status
Tracking

Please register or sign in to reply

0 Assignees

None

Select assignees

0 Reviewers

None

Request review from

Labels

cla-signed monitoring team/code-intelligence team/language-platform-and-navigation

Select labels

Manage project labels

Milestone

None

Time tracking

Participants

There are currently no pipelines.

To run a merge request pipeline, the jobs in the CI/CD configuration file must be configured to run in merge request pipelines and you must have sufficient permissions in the source project.