monitoring: add networking dashboard for Zoekt pods
Created by: ggilmore
This PR adds a new dashboard for Zoekt that displays various networking metrics (transmission/receive rate, errors, dropped packets).
It also includes two other minor changes:
- better documentation for how to use the monitoring stack while using a dataset from another instance (like k8s.sgdev.org)
- tracked down and fixed a bug in our monitoring generator that caused our template variables (that we use for per-instance metrics) to be much more permissive than intended.
Notes:
- This section is hidden by default.
- Networking metrics are only available at the pod level, not the container level (all containers in a pod share networking resources [ip, ports, etc.] - see the original k8s networking design docs for more on this in the "container to container" and "pod to pod" sections. For us, this means that the networking metrics for
zoekt-indexserver
andzoekt-webserver
are aggregated together. Given this, how do y'all feel about copying this same set of dashboards onzoekt-webserver
's Grafana page)? - These dashboards only work on Kubernetes instances for now. The labels they rely on are only provided in kubernetes deployments, there is a different set of labels in docker-compose deployments. I think that's possible to get it working on both environments by looking into label rewriting, but I figured this wasn't a blocker for now. (gitserver's metrics have the same limitation).