Skip to content

internal/tracer: add OpenTelemetry tracer

Warren Gifford requested to merge tracer-opentelemetry into main

Created by: bobheadxi

This approach should "just work". We extend internal/tracer to support a new tracer type, opentelemetry, that sends everything in OpenTelemetry format (hopefully correctly) using the OpenTracing bridge, which means it looks like no code changes are necessary!

Took me a while a very long time to wrap my head around, but this is how our tracing implementation seems to work:

  1. internal/tracer.Init() -> sets a switchableTracer to opentracing.GlobalTracer()
  2. Everyone else uses opentracing.GlobalTracer -> switchableTracer
  3. We hot-swap the opentracing.Tracer inside switchableTracer with whatever is in site config

The OTel collector host is configurable via OTEL_EXPORTER_OTLP_ENDPOINT - a clunky name, but one that shows up in the wild frequently, and OTel export can be enabled by setting:

  "observability.tracing": {
    "sampling": "selective",
    "type": "opentelemetry",
  },

See below + test plan for more details.

Closes https://github.com/sourcegraph/sourcegraph/issues/27386

Incomplete but IMO should not be blocking, because this tracing implementation is not used in practice anywhere yet:

  • Docs (intentional omission for now, until we've had more time to test this in practice). I have included some very minimal dev docs however
  • Spans that cross to zoekt-webserver - I can see the individual spans, but they aren't linked properly in Jaeger, see below (left is before, right is after)
Before After

I suspect this is because I haven't configured the propagator correctly to work with pure OpenTracing implementations like Zoekt, but after some digging I haven't been able to figure this out. Note that spans that go between Sourcegraph services work fine:

image

Filed the above as a follow-up in https://github.com/sourcegraph/sourcegraph/issues/38232

Test plan

  1. Set dev-private site config to use opentelemetry
  2. sg start otel -> runs otel-collector and jaeger
  3. sg start
  4. Run a complex query with &trace=1, e.g. https://sourcegraph.test:3443/search?q=context%3Aglobal+foobar%28...%29&patternType=structural&trace=1

Quick demo: https://www.loom.com/share/a8f86adfac984f6085a000f1dc6ba4c7

Merge request reports