observation: integrate Logger into all levels of Observation

Created by: bobheadxi

Creating and tagging operations now create loggers embedded into the various Observation types that can be used directly for ✨ structured logs ✨ . Closes https://github.com/sourcegraph/sourcegraph/issues/34217

This makes a few API changes, hence the extensive diff - the main thing to review is internal/observation/observation.go

• Diff: https://github.com/sourcegraph/sourcegraph/pull/34456/files#diff-e53a07f3674b9472217df0b8970dd39a67f4f884b3f1e61e2e7a01ee77aaf526
• Full thing: https://github.com/sourcegraph/sourcegraph/blob/observation-logger/internal/observation/observation.go

Notes:

1. observation.Operation can no longer be used in init() because we instantiate a logger if one is not present - there were a few usages that I've removed/migrated to sync.Once
2. I've replaced With with WithAndLogger, which encourages using TraceLogger to do logging, and the previous With would just call WithAndLogger and discard the value if unused anyway so might as well discard at the callsite

Example:

    observationContext := observation.Context{
        Logger:     log.Scoped("my-scope", "a simple description"),
        Tracer:     &trace.Tracer{Tracer: opentracing.GlobalTracer()},
        Registerer: prometheus.DefaultRegisterer,
    }

    operation := observationContext.Operation(observation.Op{
        Name:         "Thing.SomeOperation",
        MetricLabels: []string{"some_operation"},
        Metrics:      metrics,
    })

    // You can log some logs directly using operation - these logs will be structured
    // with context about your operation.
    operation.Info("something happened!", log.String("additional", "context"))

    function SomeOperation(ctx context.Context) (err error) {
        // logs and metrics may be available before or after the operation, so they
        // can be supplied either at the start of the operation, or after in the
        // defer of endObservation.

        ctx, trace, endObservation := operation.With(ctx, &err, observation.Args{ /* logs and metrics */ })
        defer func() { endObservation(1, observation.Args{ /* additional logs and metrics */ }) }()

        // ...

        // You can log some logs directly from the returned trace - these logs will be
        // structured with the trace ID, trace fields, and observation context.
        trace.Info("I did the thing!", log.Int("things", 3))

        // ...
    }

Test plan

sg start and sg start oss works without a hitch, tests pass

Scope naming can probably use some refining, but I'll let everyone tweak it as they see fit in follow-ups :)