monitoring: dashboards reference, refresh observability docs

Created by: bobheadxi

Closes https://github.com/sourcegraph/sourcegraph/issues/15872

Key things to review

A lot of the diff is generated. Commit-by-commit review should work too, or:

• Code
  • docs generation code
  • observable API, dashboard generation, and observable validation
• Documentation
  • metrics
  • generator overview
• Output - go generate ./monitoring && ./dev/docsite.sh -config doc/docsite.json serve -http=localhost:5080, then:
  • http://localhost:5080/admin/observability
  • http://localhost:5080/admin/observability/dashboards
  • http://localhost:5080/admin/observability/alert_solutions

Also see screenshots below

Changes

• Generated dashboards.md reference
  • All dashboards have an entry here to avoid confusion
  • Links to/from alert solutions where relevant
  • See below
• New field: Observable.Interpretation
  • Required if no alerts, optional if set
  • Supposed to be looser reference than PossibleSolutions
  • Can be set to "none" - I've set most existing ones to none, and converted those with dangling PossibleSolutions into new format
• Improved validation for monitoring definitions, e.g. PossibleSolutions without alerts and alerts with NoAlert: true now emit errors, and the output is improved
• Updated documentation (observability/index.md, metrics.md, etc.) for cleanup + reference the new dashboards docs - mostly fb5284a77aac7988f0d8d5ad6d4e9bcc303c642c
  • Reorder metrics page for relevance - highlight Grafana first, then configuration, then direct access (most deployments shouldn't need direct access instructions). Also updated the direct access docs for consistency. Collapse metrics guide page since there's not much to warrant a separate page here it seems.
  • Clean up observability homepage, break out sections into new pages. Add links to our generated references. See below
  • More highlights for monitoring docs (1dfbc144d462413ea52b2fd26fd68f0c99c55fcb)
• Links to relevant references (panel reference and alert solutions) in Grafana:

Panel documentation example:

Reference to panel documentation in alert solutions:

Updated observability homepage: