simple traces for search

Created by: keegancsmith

Problem

Tracing is a great way for us to narrow down why a search is slow. We have jaeger (and opentracing in general) which gives us great distributed tracing. However, not all customers set this up. We have net/trace on all services, but only our single docker image makes it convenient to access (see https://github.com/sourcegraph/sourcegraph/issues/8654). We are missing a convenient way to understand the breakdown of what happened in a specific request.

Proposal

Add some in-process tracing for search in sourcegraph-frontend. A field debugLog: string added to the search graphQL resolver. When requested we return debug logs. The browser can then request that field and console.log it in the response. Additionally also log it in the sourcegraph-frontend in the cases of no response (eg timeouts/OOM).

Note: Given this is opt-in, we don't need to be overly concerned about performance. IE we don't need to do crazy sync/sampling stuff.

Note: There is potential to leak sensitive data. The interface of ./internal/trace should contains a way to indicate sensitive vs non-sensitive data. We can then decide to show sensitive bits if the user is an admin.

Optional: Hook into ./internal/trace pkg

We don't need to add in logs, instead we can hook into ./internal/trace. I recommend this route. Concerns are around potentially large responses. For example we may trace an event per repo when using searcher. If so we should look into limiting what we capture for a request.

Optional: URL param toggle

Settings ?trace=1 automatically turns on tracing for a request. We can inspect the referer header (or maybe also look at a cookie). This is how it has worked at previous companies I have been at.