SSBC-EM2.1: Malo can use SSBC in a demo

Created by: mrnugget

Experimental scope:

As a customer, I want to run a large batch changes, that I can parallelize on several executors so that they run faster. I'm fine with debugging with the CLI first. All I want to do is paste a spec, run it across executors, be able to see the logs per repo/step, then go to preview/apply. I also want to be able to edit the spec and re-run it to correct mistakes.

Items:

• One executor job per workspace (repository+path+commit)
  • src-cli able to execute a batch spec in a given workspace (not doing resolving of workspaces at all) https://github.com/sourcegraph/src-cli/pull/608 @mrnugget
  • Add the necessary workers in the backend to resolve a batch spec into workspaces and execute those https://github.com/sourcegraph/sourcegraph/pull/24749 @mrnugget
    • Create batch_spec_resolution_jobs
    • Create batch_spec_workspaces
    • Execute batch_spec_workspace_execution_job, one per executor
      • batch_spec_workspace_execution_job are executor-jobs etc.
      • Each workspace-execution-job, if successful, creates a changeset_spec (from the JSON log line of src-cli) with correct batch_spec_id and stores changeset_spec.id on batch_spec_workspace_execution_job.changeset_spec_id
  • Follow-up: https://github.com/sourcegraph/sourcegraph/pull/24886 @mrnugget
  • Follow-up: https://github.com/sourcegraph/sourcegraph/pull/24965 @mrnugget
  • https://github.com/sourcegraph/sourcegraph/pull/24985 @mrnugget
  • Remove old code batch_spec_executions @mrnugget
  • Cancellation of batch_spec_workspace_execution_jobs https://github.com/sourcegraph/sourcegraph/pull/24893 @mrnugget
  • Build and switch frontend UI to use new API @eseliger
    • API schema and stub resolvers for SSBC @eseliger https://github.com/sourcegraph/sourcegraph/pull/24966
    • Implement createBatchSpecFromRaw/replaceBatchSpecInput/executeBatchSpec mutations @mrnugget
    • Implement workspace resolvers
    • Fix the TODOs left in https://github.com/sourcegraph/sourcegraph/pull/25182/files
    • Implement rest of mutations...
    • Move parsing of log lines to backend
    • UI: build workflow to create, preview, and then run a batch spec
  • Clean up non-executed workspaces, resolution_jobs. Update: We don't need to do this, since we want to keep batch specs around for now and use replaceBatchSpecInput to not amass a large number of specs.
  • Switch dogfood executor to new queue @eseliger
  • result from new graphql api: implement validatechangesetspecs https://github.com/sourcegraph/sourcegraph/pull/25547 @mrnugget
  • Implement "importingChangesets" to restore functionality. First version: in request, when creating batch spec, simply create imported changesets. @mrnugget https://github.com/sourcegraph/sourcegraph/pull/25397
  • Ensure that only one batch_spec_resolution_job exists for the batch spec at a given time @mrnugget
  • https://github.com/sourcegraph/sourcegraph/issues/25943 @mrnugget
• Alpha customer deployment of executor @eseliger
  • Improve deployment of executor
    • Build AMI in CI for executor https://github.com/sourcegraph/sourcegraph/pull/23364
    • Implement AWS support @eseliger
    • Deploy new executor from CI
    • Generalize code-intel terraform stuff in infra repo https://github.com/sourcegraph/infrastructure/pull/2704
  • Documentation for customers on how to deploy executor @eseliger
    • Bare metal
    • Google Cloud
    • AWS
    • Docker registry mirror
  • Publish and generalize terraform files for executor on Google Cloud
    • Same for metrics scraping into Prometheus
  • Publish and generalize terraform files for executor on AWS
    • Same for metrics scraping into Prometheus
• Get rid of technical debt in authentication tokens before shipping https://github.com/sourcegraph/sourcegraph/pull/25638 @mrnugget