Skip to content

ci: add automatic retries when agent is lost

Warren Gifford requested to merge main-dry-run/devx/retry-agent-lost into main

Created by: jhchabran

This tweaks the pipeline to automatically insert a retry when an agent is lost, which we can detect by seeing that the exit status is set to -1.

Test plan

Main dry run, no changes in behaviour unless agent dies. Killed an agent on purpose and should see the job being restarted.

image

If manual retry is disabled, we don't restart it either:

    command:
    - ./tr dev/ci/yarn-web-integration.sh "client/web/src/integration/batches.test.ts
      client/web/src/integration/blob-viewer.test.ts"
    depends_on:
    - puppeteer:prep
    # ...
    key: puppeteerelectricplugPuppeteertestschunk1
    label: ':puppeteer::electric_plug: Puppeteer tests chunk #1'
    # ...
    retry:
      manual:
        allowed: false
        reason: The Percy build is not finalized if one of the concurrent agents fails.
          To retry correctly, restart the entire pipeline.
    timeout_in_minutes: "60"

Merge request reports