dev/ci: grouped operations
Created by: bobheadxi
Integrate support for Buildkite group steps within our operation.Set abstraction by treating named sets as another Pipeline, and moves our existing steps into appropriate named sets.
This PR also includes some minor cleanup
Example: https://buildkite.com/sourcegraph/sourcegraph/builds/127901
sg ci preview
❯ sg ci preview
If the current branch were to be pushed, the following pipeline would be run:
Detected run type:
Main dry run
Detected changed files (8):
.buildkite/pipeline.yml
enterprise/dev/ci/gen-pipeline.go
enterprise/dev/ci/internal/buildkite/buildkite.go
enterprise/dev/ci/internal/ci/changed/changed.go
enterprise/dev/ci/internal/ci/operations.go
enterprise/dev/ci/internal/ci/operations/operations.go
enterprise/dev/ci/internal/ci/pipeline.go
enterprise/dev/ci/internal/ci/runtype.go
Detected changes:
Affects CI scripts: false
Affects Terraform: false
Affects ExecutorDockerRegistryMirror: false
Affects Go: true
Affects Client: false
Affects Docs: false
Affects Dockerfiles: false
Affects GraphQL: false
Computed build steps:
Pipeline setup
:snail: Trigger async
Image builds
:docker: :construction: alpine-3.12
:docker: :construction: alpine-3.14
:docker: :construction: cadvisor
:docker: :construction: codeinsights-db
:docker: :construction: codeintel-db
:docker: :construction: frontend
:docker: :construction: github-proxy
:docker: :construction: gitserver
:docker: :construction: grafana
:docker: :construction: indexed-searcher
:docker: :construction: jaeger-agent
:docker: :construction: jaeger-all-in-one
:docker: :construction: minio
:docker: :construction: postgres-12.6-alpine
:docker: :construction: postgres_exporter
:docker: :construction: precise-code-intel-worker
:docker: :construction: prometheus
:docker: :construction: redis-cache
:docker: :construction: redis-store
:docker: :construction: redis_exporter
:docker: :construction: repo-updater
:docker: :construction: search-indexer
:docker: :construction: searcher
:docker: :construction: symbols
:docker: :construction: syntax-highlighter
:docker: :construction: worker
:docker: :construction: migrator
:docker: :construction: server
:packer: :construction: executor image
Image security scans
:trivy: :docker: :mag: "alpine-3.12"
→ depends on alpine-3-12:candidate
:trivy: :docker: :mag: "alpine-3.14"
→ depends on alpine-3-14:candidate
:trivy: :docker: :mag: "cadvisor"
→ depends on cadvisor:candidate
:trivy: :docker: :mag: "codeinsights-db"
→ depends on codeinsights-db:candidate
:trivy: :docker: :mag: "codeintel-db"
→ depends on codeintel-db:candidate
:trivy: :docker: :mag: "frontend"
→ depends on frontend:candidate
:trivy: :docker: :mag: "github-proxy"
→ depends on github-proxy:candidate
:trivy: :docker: :mag: "gitserver"
→ depends on gitserver:candidate
:trivy: :docker: :mag: "grafana"
→ depends on grafana:candidate
:trivy: :docker: :mag: "indexed-searcher"
→ depends on indexed-searcher:candidate
:trivy: :docker: :mag: "jaeger-agent"
→ depends on jaeger-agent:candidate
:trivy: :docker: :mag: "jaeger-all-in-one"
→ depends on jaeger-all-in-one:candidate
:trivy: :docker: :mag: "minio"
→ depends on minio:candidate
:trivy: :docker: :mag: "postgres-12.6-alpine"
→ depends on postgres-12-6-alpine:candidate
:trivy: :docker: :mag: "postgres_exporter"
→ depends on postgres_exporter:candidate
:trivy: :docker: :mag: "precise-code-intel-worker"
→ depends on precise-code-intel-worker:candidate
:trivy: :docker: :mag: "prometheus"
→ depends on prometheus:candidate
:trivy: :docker: :mag: "redis-cache"
→ depends on redis-cache:candidate
:trivy: :docker: :mag: "redis-store"
→ depends on redis-store:candidate
:trivy: :docker: :mag: "redis_exporter"
→ depends on redis_exporter:candidate
:trivy: :docker: :mag: "repo-updater"
→ depends on repo-updater:candidate
:trivy: :docker: :mag: "search-indexer"
→ depends on search-indexer:candidate
:trivy: :docker: :mag: "searcher"
→ depends on searcher:candidate
:trivy: :docker: :mag: "symbols"
→ depends on symbols:candidate
:trivy: :docker: :mag: "syntax-highlighter"
→ depends on syntax-highlighter:candidate
:trivy: :docker: :mag: "worker"
→ depends on worker:candidate
:trivy: :docker: :mag: "migrator"
→ depends on migrator:candidate
:trivy: :docker: :mag: "server"
→ depends on server:candidate
Linters and static analysis
:lipstick: Prettier
:clipboard: Misc Linters
:lipstick: :graphql: GraphQL lint
:docker: Lint
:lock: security - checkov
:memo: Check and build docsite
Client and GraphQL checks
:puppeteer::electric_plug: Puppeteer tests prep
:puppeteer::electric_plug: Puppeteer tests chunk #1
→ depends on puppeteer:prep
:puppeteer::electric_plug: Puppeteer tests chunk #2
→ depends on puppeteer:prep
:puppeteer::electric_plug: Puppeteer tests chunk #3
→ depends on puppeteer:prep
:puppeteer::electric_plug: Puppeteer tests chunk #4
→ depends on puppeteer:prep
:puppeteer::electric_plug: Puppeteer tests chunk #5
→ depends on puppeteer:prep
:puppeteer::electric_plug: Puppeteer tests chunk #6
→ depends on puppeteer:prep
:puppeteer::electric_plug: Puppeteer tests chunk #7
→ depends on puppeteer:prep
:puppeteer::electric_plug: Puppeteer tests chunk #8
→ depends on puppeteer:prep
:puppeteer::electric_plug: Puppeteer tests chunk #9
→ depends on puppeteer:prep
:puppeteer::electric_plug: Puppeteer tests finalize
→ depends on puppeteer:chunk:1, puppeteer:chunk:2, puppeteer:chunk:3, puppeteer:chunk:4, puppeteer:chunk:5, ... (4 more steps)
:chromatic: Upload Storybook to Chromatic
:jest: Test shared client code
:jest: Test wildcard client code
:webpack::globe_with_meridians: Build
:webpack::globe_with_meridians::moneybag: Enterprise build
:jest::globe_with_meridians: Test
:chrome: Puppeteer tests for chrome extension
:jest::chrome: Test browser extension
:jest: Test branded client code
:eslint: Typescript eslint
:stylelint: Stylelint
Go and GraphQL checks
:go: Test (all)
:go: Test (enterprise/internal/codeintel/stores/dbstore)
:go: Test (enterprise/internal/codeintel/stores/lsifstore)
:go: Test (enterprise/internal/insights)
:go: Test (internal/database)
:go: Test (internal/repos)
:go: Test (enterprise/internal/batches)
:go: Test (cmd/frontend)
:go: Test (enterprise/internal/database)
:go: Test (enterprise/cmd/frontend/internal/batches/resolvers)
:go: Build
Database backcompat tests
:go::postgres: Backcompat test (all)
:go::postgres: Backcompat test (enterprise/internal/codeintel/stores/dbstore)
:go::postgres: Backcompat test (enterprise/internal/codeintel/stores/lsifstore)
:go::postgres: Backcompat test (enterprise/internal/insights)
:go::postgres: Backcompat test (internal/database)
:go::postgres: Backcompat test (internal/repos)
:go::postgres: Backcompat test (enterprise/internal/batches)
:go::postgres: Backcompat test (cmd/frontend)
:go::postgres: Backcompat test (enterprise/internal/database)
:go::postgres: Backcompat test (enterprise/cmd/frontend/internal/batches/resolvers)
CI scripts tests
:bash: test-trace-command.sh
Integration tests
:chains: Backend integration tests
→ depends on server:candidate
:docker::brain: Code Intel QA
→ depends on server:candidate
:chromium: Sourcegraph E2E
→ depends on server:candidate
:docker::chromium: Sourcegraph QA
→ depends on server:candidate
:docker::arrow_double_up: Sourcegraph Upgrade
→ depends on server:candidate
Publish images
:docker: :truck: alpine-3.12
:docker: :truck: alpine-3.14
:docker: :truck: cadvisor
:docker: :truck: codeinsights-db
:docker: :truck: codeintel-db
:docker: :truck: frontend
:docker: :truck: github-proxy
:docker: :truck: gitserver
:docker: :truck: grafana
:docker: :truck: indexed-searcher
:docker: :truck: jaeger-agent
:docker: :truck: jaeger-all-in-one
:docker: :truck: minio
:docker: :truck: postgres-12.6-alpine
:docker: :truck: postgres_exporter
:docker: :truck: precise-code-intel-worker
:docker: :truck: prometheus
:docker: :truck: redis-cache
:docker: :truck: redis-store
:docker: :truck: redis_exporter
:docker: :truck: repo-updater
:docker: :truck: search-indexer
:docker: :truck: searcher
:docker: :truck: symbols
:docker: :truck: syntax-highlighter
:docker: :truck: worker
:docker: :truck: migrator
:docker: :truck: server
:arrow_heading_up: Uploading trace to HoneyComb