distribution: add monitoring architecture page

Created by: slimsag

@bobheadxi noted to me over Slack:

I think this is a great idea and something that I have long intended to put together - but I think an RFC may be the wrong forum for it because many of these decisions have already been made. I am therefor proposing the following:

1. The current monitoring architecture and decisions we have already made are documented and live in this document. For any question @pecigonzalo has asked me or you @bobheadxi about "why are we doing this this way?" I would like this document to explain it
2. For changes to the current monitoring architecture and decisions we are considering, those should be RFCs (or, if small changes they can be lightweight Proposal: ... GitHub issues, like https://github.com/sourcegraph/sourcegraph/issues/12010).

@bobheadxi can you own filling this out from here with all the context I've transferred to you, any questions Gonza has asked you, etc.? And then I'll follow-up and add any context I may have as well after

Rendered: https://github.com/sourcegraph/about/blob/monitoring-architecture/handbook/engineering/distribution/observability/monitoring_architecture.md