all: log traceid in all HTTP handlers
Created by: tsenart
I am experimenting with using Grafana Cloud for traces and logs. Grafana Tempo, unlike Jaeger, can store a much higher amount of traces because it doesn't index them and uses object storage, which means we can see traces for requests that happened weeks in the past, with much less sampling.
However, we need to know a trace id to see a trace. Trace discovery through logs is the standard approach. Trace discovery through metric examplars can be set-up in the future (i.e. you click on a a few "examples" in a panel's graph).
This changeset leverages our existing trace.HTTPTraceMiddleware
in every service to log a traceid, not just frontend, and changes the default logging level to info rather than debug. The only concern I have is that the dev env can become quite noisy due to all the background polling that services do. Should we keep the debug log level, and set that level in production only? Seems off somehow.