Use net/http/httptrace to trace HTTP client requests
Created by: beyang
Problem: Some customers experience network connectivity issues between different services in Sourcegraph. In debugging issue where requests from service X to service Y fail, we don't have an easy way to determine if the request is at all making it to service Y (vs. the request makes it to service Y and some error occurs in service Y, or the request never leaves service X due to an error that happens in service X).
Proposed solution: Add https://golang.org/pkg/net/http/httptrace/ to our HTTP clients throughout the Sourcegraph backend. This will ensure there is a span corresponding to each HTTP client request in a trace, which can be used to determine if the issue is due to network connectivity or something else. (E.g., if a span exists for the HTTP client request, but not the corresponding HTTP handler in the receiving service, that implies that the error lies between the HTTP client and the HTTP server.)
More context in this Slack thread.