Skip to content

monitoring: update vision, revamp pillars

Warren Gifford requested to merge monitoring-pillars-redux into main

Created by: bobheadxi

Given the recent (and very welcome!) drive to improve how monitoring works for engineers at Sourcegraph, I thought I might put up a PR to start shaping those discussions into formal guidelines that we can reference when driving changes to the monitoring generator!

Changes

The overall goal of this PR is to relax our guidelines to enable more flexibility to make changes to the tooling without introducing "incompatibilities" with the pillars, as well as to allow everyone more freedom to make alternative decisions based on their best judgement to a greater extent.

  • Add a new long-term vision item: serving Sourcegraph engineers!
  • "Five pillars" to just "pillars": we should be able to add or remove guidelines as Sourcegraph changes.
  • In general, build our pillars on more relaxed, positive messaging ("Should...") instead of negative ("...is forbidden") - this hopefully makes the reasoning behind tooling restrictions more palpable, and add more flexibility.
  • Each pillar can now include an "Exceptions" section.
  • Some pillars have been changed or removed.

Would appreciate if each team who currently has alerts configured could take a look!

Rendered: https://github.com/sourcegraph/about/blob/monitoring-pillars-redux/handbook/engineering/observability/monitoring_pillars.md

Follow-up implementation tasks

From here, we can create issues to implement improvements to the generator to align it with these new pillars. This work will be tracked in the monitoring redux project, which the Distribution team is hoping to prioritize soon.

Merge request reports