WIP Proposal: Remove unneeded metrics

Created by: uwedeportivo

prom-metrics-doc generates a list of all metrics with corresponding documentation.

We have a couple of instances where we use counters and histograms for the same event. Histograms have an internal counter we can use and get rid of the additional counter.

Inventory of metrics related to http requests

pkg/metrics/metrics.go

Used to measure requests to external services (bitbucket, github, etc) from the client POV (http client instrumentation). Currently consists of counter vector (breakdown by http status code + special case for error). PR 5093 adds request duration histogram vector.

Example metrics: repo-updater (this includes PR 5093)

# HELP src_github_request_duration_seconds Time (in seconds) spent on request.
# TYPE src_github_request_duration_seconds histogram
src_github_request_duration_seconds_bucket{category="graphql",code="200",le="0.005"} 0
src_github_request_duration_seconds_bucket{category="graphql",code="200",le="0.01"} 0
...
src_github_request_duration_seconds_bucket{category="graphql",code="200",le="10"} 8
src_github_request_duration_seconds_bucket{category="graphql",code="200",le="+Inf"} 8
src_github_request_duration_seconds_sum{category="graphql",code="200"} 3.1095035149999997
src_github_request_duration_seconds_count{category="graphql",code="200"} 8
# HELP src_github_requests_total Total number of requests sent to the GitHub API.
# TYPE src_github_requests_total counter
src_github_requests_total{category="graphql",code="200"} 8

cmd/frontend/backend/trace.go

Used by cmd/frontend/backend/repos.go (for example func (s *repos) AddGitHubDotComRepository ).

Example metrics: frontend

src_backend_client_request_duration_seconds_bucket{method="Repos.GetByName",success="false",le="0.2"} 5
src_backend_client_request_duration_seconds_bucket{method="Repos.GetByName",success="false",le="0.5"} 5
src_backend_client_request_duration_seconds_bucket{method="Repos.GetByName",success="false",le="1"} 5
src_backend_client_request_duration_seconds_bucket{method="Repos.GetByName",success="false",le="2"} 5

pkg/trace/httptrace.go

Woven into trace.Middleware(). Used only by cmd/frontend/internal/cli.newExternalHTTPHandler() to create the external http handler for frontend.

Example metrics: frontend

src_http_request_duration_seconds_bucket{code="200",method="get",repo="unknown",route="search",le="2"} 1
src_http_request_duration_seconds_bucket{code="200",method="get",repo="unknown",route="search",le="5"} 1
src_http_request_duration_seconds_bucket{code="200",method="get",repo="unknown",route="search",le="10"} 1
src_http_request_duration_seconds_bucket{code="200",method="get",repo="unknown",route="search",le="30"} 1
src_http_request_duration_seconds_bucket{code="200",method="get",repo="unknown",route="search",le="+Inf"} 1

cmd/repo-updater/repoupdater/observability.go

Created by repoupdater.NewHandlerMetrics() wraps handlers (server POV in repo-updater).

Example metrics: repo-updater

src_repoupdater_http_handler_duration_seconds_bucket{code="200",path="/status-messages",le="0.01"} 2150
src_repoupdater_http_handler_duration_seconds_bucket{code="200",path="/status-messages",le="0.025"} 2150
src_repoupdater_http_handler_duration_seconds_bucket{code="200",path="/status-messages",le="0.05"} 2150
src_repoupdater_http_handler_duration_seconds_bucket{code="200",path="/status-messages",le="0.1"} 2150
src_repoupdater_http_handler_duration_seconds_bucket{code="200",path="/status-messages",le="0.25"} 2150
src_repoupdater_http_handler_duration_seconds_bucket{code="200",path="/status-messages",le="0.5"} 2150