Skip to content

dev/ci: grouped operations

Warren Gifford requested to merge main-dry-run/grouped-operations into main

Created by: bobheadxi

Integrate support for Buildkite group steps within our operation.Set abstraction by treating named sets as another Pipeline, and moves our existing steps into appropriate named sets.

This PR also includes some minor cleanup

Example: https://buildkite.com/sourcegraph/sourcegraph/builds/127901

image

sg ci preview
❯ sg ci preview                               
If the current branch were to be pushed, the following pipeline would be run:
Detected run type:
        Main dry run
Detected changed files (8):
        .buildkite/pipeline.yml
        enterprise/dev/ci/gen-pipeline.go
        enterprise/dev/ci/internal/buildkite/buildkite.go
        enterprise/dev/ci/internal/ci/changed/changed.go
        enterprise/dev/ci/internal/ci/operations.go
        enterprise/dev/ci/internal/ci/operations/operations.go
        enterprise/dev/ci/internal/ci/pipeline.go
        enterprise/dev/ci/internal/ci/runtype.go
Detected changes:
        Affects CI scripts: false
        Affects Terraform: false
        Affects ExecutorDockerRegistryMirror: false
        Affects Go: true
        Affects Client: false
        Affects Docs: false
        Affects Dockerfiles: false
        Affects GraphQL: false
Computed build steps:
        Pipeline setup
                :snail: Trigger async
        Image builds
                :docker: :construction: alpine-3.12
                :docker: :construction: alpine-3.14
                :docker: :construction: cadvisor
                :docker: :construction: codeinsights-db
                :docker: :construction: codeintel-db
                :docker: :construction: frontend
                :docker: :construction: github-proxy
                :docker: :construction: gitserver
                :docker: :construction: grafana
                :docker: :construction: indexed-searcher
                :docker: :construction: jaeger-agent
                :docker: :construction: jaeger-all-in-one
                :docker: :construction: minio
                :docker: :construction: postgres-12.6-alpine
                :docker: :construction: postgres_exporter
                :docker: :construction: precise-code-intel-worker
                :docker: :construction: prometheus
                :docker: :construction: redis-cache
                :docker: :construction: redis-store
                :docker: :construction: redis_exporter
                :docker: :construction: repo-updater
                :docker: :construction: search-indexer
                :docker: :construction: searcher
                :docker: :construction: symbols
                :docker: :construction: syntax-highlighter
                :docker: :construction: worker
                :docker: :construction: migrator
                :docker: :construction: server
                :packer: :construction: executor image
        Image security scans
                :trivy: :docker: :mag: "alpine-3.12"
                        → depends on alpine-3-12:candidate
                :trivy: :docker: :mag: "alpine-3.14"
                        → depends on alpine-3-14:candidate
                :trivy: :docker: :mag: "cadvisor"
                        → depends on cadvisor:candidate
                :trivy: :docker: :mag: "codeinsights-db"
                        → depends on codeinsights-db:candidate
                :trivy: :docker: :mag: "codeintel-db"
                        → depends on codeintel-db:candidate
                :trivy: :docker: :mag: "frontend"
                        → depends on frontend:candidate
                :trivy: :docker: :mag: "github-proxy"
                        → depends on github-proxy:candidate
                :trivy: :docker: :mag: "gitserver"
                        → depends on gitserver:candidate
                :trivy: :docker: :mag: "grafana"
                        → depends on grafana:candidate
                :trivy: :docker: :mag: "indexed-searcher"
                        → depends on indexed-searcher:candidate
                :trivy: :docker: :mag: "jaeger-agent"
                        → depends on jaeger-agent:candidate
                :trivy: :docker: :mag: "jaeger-all-in-one"
                        → depends on jaeger-all-in-one:candidate
                :trivy: :docker: :mag: "minio"
                        → depends on minio:candidate
                :trivy: :docker: :mag: "postgres-12.6-alpine"
                        → depends on postgres-12-6-alpine:candidate
                :trivy: :docker: :mag: "postgres_exporter"
                        → depends on postgres_exporter:candidate
                :trivy: :docker: :mag: "precise-code-intel-worker"
                        → depends on precise-code-intel-worker:candidate
                :trivy: :docker: :mag: "prometheus"
                        → depends on prometheus:candidate
                :trivy: :docker: :mag: "redis-cache"
                        → depends on redis-cache:candidate
                :trivy: :docker: :mag: "redis-store"
                        → depends on redis-store:candidate
                :trivy: :docker: :mag: "redis_exporter"
                        → depends on redis_exporter:candidate
                :trivy: :docker: :mag: "repo-updater"
                        → depends on repo-updater:candidate
                :trivy: :docker: :mag: "search-indexer"
                        → depends on search-indexer:candidate
                :trivy: :docker: :mag: "searcher"
                        → depends on searcher:candidate
                :trivy: :docker: :mag: "symbols"
                        → depends on symbols:candidate
                :trivy: :docker: :mag: "syntax-highlighter"
                        → depends on syntax-highlighter:candidate
                :trivy: :docker: :mag: "worker"
                        → depends on worker:candidate
                :trivy: :docker: :mag: "migrator"
                        → depends on migrator:candidate
                :trivy: :docker: :mag: "server"
                        → depends on server:candidate
        Linters and static analysis
                :lipstick: Prettier
                :clipboard: Misc Linters
                :lipstick: :graphql: GraphQL lint
                :docker: Lint
                :lock: security - checkov
                :memo: Check and build docsite
        Client and GraphQL checks
                :puppeteer::electric_plug: Puppeteer tests prep
                :puppeteer::electric_plug: Puppeteer tests chunk #1
                        → depends on puppeteer:prep
                :puppeteer::electric_plug: Puppeteer tests chunk #2
                        → depends on puppeteer:prep
                :puppeteer::electric_plug: Puppeteer tests chunk #3
                        → depends on puppeteer:prep
                :puppeteer::electric_plug: Puppeteer tests chunk #4
                        → depends on puppeteer:prep
                :puppeteer::electric_plug: Puppeteer tests chunk #5
                        → depends on puppeteer:prep
                :puppeteer::electric_plug: Puppeteer tests chunk #6
                        → depends on puppeteer:prep
                :puppeteer::electric_plug: Puppeteer tests chunk #7
                        → depends on puppeteer:prep
                :puppeteer::electric_plug: Puppeteer tests chunk #8
                        → depends on puppeteer:prep
                :puppeteer::electric_plug: Puppeteer tests chunk #9
                        → depends on puppeteer:prep
                :puppeteer::electric_plug: Puppeteer tests finalize
                        → depends on puppeteer:chunk:1, puppeteer:chunk:2, puppeteer:chunk:3, puppeteer:chunk:4, puppeteer:chunk:5, ... (4 more steps)
                :chromatic: Upload Storybook to Chromatic
                :jest: Test shared client code
                :jest: Test wildcard client code
                :webpack::globe_with_meridians: Build
                :webpack::globe_with_meridians::moneybag: Enterprise build
                :jest::globe_with_meridians: Test
                :chrome: Puppeteer tests for chrome extension
                :jest::chrome: Test browser extension
                :jest: Test branded client code
                :eslint: Typescript eslint
                :stylelint: Stylelint
        Go and GraphQL checks
                :go: Test (all)
                :go: Test (enterprise/internal/codeintel/stores/dbstore)
                :go: Test (enterprise/internal/codeintel/stores/lsifstore)
                :go: Test (enterprise/internal/insights)
                :go: Test (internal/database)
                :go: Test (internal/repos)
                :go: Test (enterprise/internal/batches)
                :go: Test (cmd/frontend)
                :go: Test (enterprise/internal/database)
                :go: Test (enterprise/cmd/frontend/internal/batches/resolvers)
                :go: Build
        Database backcompat tests
                :go::postgres: Backcompat test (all)
                :go::postgres: Backcompat test (enterprise/internal/codeintel/stores/dbstore)
                :go::postgres: Backcompat test (enterprise/internal/codeintel/stores/lsifstore)
                :go::postgres: Backcompat test (enterprise/internal/insights)
                :go::postgres: Backcompat test (internal/database)
                :go::postgres: Backcompat test (internal/repos)
                :go::postgres: Backcompat test (enterprise/internal/batches)
                :go::postgres: Backcompat test (cmd/frontend)
                :go::postgres: Backcompat test (enterprise/internal/database)
                :go::postgres: Backcompat test (enterprise/cmd/frontend/internal/batches/resolvers)
        CI scripts tests
                :bash: test-trace-command.sh
        Integration tests
                :chains: Backend integration tests
                        → depends on server:candidate
                :docker::brain: Code Intel QA
                        → depends on server:candidate
                :chromium: Sourcegraph E2E
                        → depends on server:candidate
                :docker::chromium: Sourcegraph QA
                        → depends on server:candidate
                :docker::arrow_double_up: Sourcegraph Upgrade
                        → depends on server:candidate
        Publish images
                :docker: :truck: alpine-3.12
                :docker: :truck: alpine-3.14
                :docker: :truck: cadvisor
                :docker: :truck: codeinsights-db
                :docker: :truck: codeintel-db
                :docker: :truck: frontend
                :docker: :truck: github-proxy
                :docker: :truck: gitserver
                :docker: :truck: grafana
                :docker: :truck: indexed-searcher
                :docker: :truck: jaeger-agent
                :docker: :truck: jaeger-all-in-one
                :docker: :truck: minio
                :docker: :truck: postgres-12.6-alpine
                :docker: :truck: postgres_exporter
                :docker: :truck: precise-code-intel-worker
                :docker: :truck: prometheus
                :docker: :truck: redis-cache
                :docker: :truck: redis-store
                :docker: :truck: redis_exporter
                :docker: :truck: repo-updater
                :docker: :truck: search-indexer
                :docker: :truck: searcher
                :docker: :truck: symbols
                :docker: :truck: syntax-highlighter
                :docker: :truck: worker
                :docker: :truck: migrator
                :docker: :truck: server
        :arrow_heading_up: Uploading trace to HoneyComb

Merge request reports

Loading