monitoring: Update syncer sync errors alerting
Created by: indradhanush
In this commit we:
- Add a Warning alert if the threshold is greater than 0.5 over 10 minutes.
- Modify the Critical alert to fire if the threshold is greater than 1 over 10 minutes.
Historically, over the last six months approximately, the value of this metric has rarely gone above 0.5 and has recovered almost immediately. Only in one instance has this gone over 1.
At the moment we get paged for this alert for what might be intermittent errors on the code host's end which get resolved immediately, reducing the signal to noise ration in our alerting.