SSBC-EM2.3: Creating 1,000 changesets is practical

Created by: malomarrec

Outcome

• we can confidently run batch changes with 1,000 changesets in SSBC
• iterating over large batch changes is at least as efficient as in the CLI

We want to test/validate:

✅ Batch spec with 500 changesets is practical:

We want to run multiple batch specs that involve 500 or more workspaces. It should be possible to feasibly get them into a state where they become applicable.

✅ Batch spec with 1000 changesets is practical

We want to run multiple batch specs that involve 1000 or more workspaces. It should be possible to feasibly get them into a state where they become applicable. The API and UI should still be responsive at that scale.

✅ Batch spec with 5000 changesets works (executed at least once)

We want to run a batch spec that involves 5000 or more workspaces. It should not crash and with a little handholding be possible to get it into an applicable state. This is the last item in this milestone and will be done once everything in here is ticked off. If it doesn't work at all, we will create follow-up items that we'll eventually address, at latest in EM3.2.

Scope

• Bug bash day @mrnugget @eseliger
• Fix bugs/problems that were discovered while testing (no estimate)
• Implement retryBatchSpecWorkspaceExecution mutation: https://github.com/sourcegraph/sourcegraph/pull/26937 (without that a 5000 changeset batch change is nearly impossible if you run into network problems) - 1d
• Implement really good server-side caching (~21d)
  • https://github.com/sourcegraph/sourcegraph/issues/26929 (~11d) (Thorsten: 6d, Erik: 4d)
  • https://github.com/sourcegraph/sourcegraph/issues/26932 (~5d)
  • https://github.com/sourcegraph/sourcegraph/issues/26933 (~0.5d remaining)
  • Bonus stretch goal, extra points: src batch exec doesn't upload changeset specs anymore and server uses cache entries to build changeset spec (~1.5d)
    • src-cli: disable building and uploading changeset specs in src batch exec 0.5d
    • In MarkComplete: find cache entry for result, check if result.Diff != "", pass into createChangesetSpecs, persist resulting changesetSpecs 1d
• MVP "Create" (Editor) page (~5d)
  • #24942 @courier-new
  • Get YAML JSONSchema support to work in monaco (for validation and hover tips) @eseliger
  • Better 3-column layout #26768 @courier-new
  • Editor resizes to remaining window height @courier-new 0.5d
  • Minimal examples/references left menu pane @courier-new 0.5d
  • Prefill editor with "Hello World" example template @courier-new <0.5d
  • Hide "Create" button from user batch changes page if you are not that user @courier-new 0.5d
  • Fix up workspaces preview scrollbar @courier-new 0.5d
  • Make workspaces preview look better (more details, visuals, making the allowUnsupported look clean) @courier-new 1d
  • Manually trigger workspaces preview (as opposed to auto-saving) Figma @courier-new 2d
• MVP "Execution" page (~4d)
  • Fix up workspaces list scrollbar @eseliger
  • Better 2-column layout 1d timebox
  • Make workspaces list look better 1d timebox
  • If only whitespace was produced, display "This step did not produce output" 0.5d
  • If the command exits with status 0 and only whitespace are produced, display "Command exited with status 0" 0.5d
  • Clean up code (less fetching, etc.) 1d