handbook: add observability developer guides

Created by: slimsag

This adds developer documentation for best-practices for developing the observability of Sourcegraph.

Right now, only documentation for monitoring is added. In the future, documentation about how we develop logging and distributed tracing should be added.

This is a first pass, and can certainly be improved in some areas -- but I have had three people ask me questions about this and be confused by the lack of documentation so I want to immediately remediate that.

Once merged, you can find these docs at https://about.sourcegraph.com/handbook/engineering/distribution/observability