Skip to content

dev/sg: instrument with OpenTelemetry traces, submit to Honeycomb

Warren Gifford requested to merge sg-otel into main

Created by: bobheadxi

Migrate sg to directly use OpenTelemetry tracing to track events, and updates sg analytics submit to submit traces to Honeycomb, everyone's favourite tracing platform.

image

Some things of note:

  • the data we end up with is quite similar to the data we had in OkayHQ, but this time as proper traces. We already had some notion of "run IDs" and durations and whatnot, so it was already quite close to tracing.
  • persisting spans to disk requires some fenangling, and is currently done through a custom "client" that sends events to disk (otlpDiskClient)
  • we provide a constructor through which spans should be created, which appends a standardized category attribute and returns a wrapped span with shortcuts for common events
  • the analytics API is a bit more complicated than before, but I think the flexibility gains here are worth it, and in practice few parts of sg will require very thorough instrumentation
  • I have not implemented an automatic way to get the Honeycomb token yet, will follow up
  • Cool side effect: you can run sg start otel, OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317, and submit your traces to a local Jaeger instance instead

Closes https://github.com/sourcegraph/sourcegraph/issues/37810

Test plan

go build -o ./sg ./dev/sg
./sg start
./sg analytics view
# get from https://team-sourcegraph.1password.com/vaults/all/allitems/skz23jc6jdi7jvve36f7nn2pla for now
export HONEYCOMB_ENV_TOKEN=...

./sg analytics submit

Some example traces:

Merge request reports