docs: Update executor observability configuration

Created by: efritz

In Configuring observability, need to give context on how to modify prometheus config in different environments.

Remaining work:

• Figure out what's required to get this running in docker-compose
• Fixup after internal testing done by Eric and Thorsten