Instrument for the question you'll ask during an incident
Good observability is built by imagining the incident before it happens: if this specific dependency fails, what question will the on-call engineer need answered in the first five minutes, and does the current instrumentation already answer it.
This is a different design exercise than generic 'add more logging' — it starts from the specific failure scenarios most likely for this particular system, not a one-size-fits-all checklist.
Three signals, deliberately connected
Logs, metrics and traces are each useful alone and far more useful connected: a trace that lets you jump directly to the relevant logs and metrics for that specific request turns a 45-minute correlation exercise into a 90-second lookup.
Investing in that connective tissue — consistent request IDs propagated across services, correlated dashboards — pays off disproportionately compared to simply adding more of any single signal type in isolation.
Treat alert quality as seriously as alert existence
An alert that fires constantly for non-issues trains the on-call team to ignore it, which is functionally worse than having no alert at all — it creates false confidence that something is being watched when, in practice, the signal has been tuned out.
Review alert precision on a regular cadence the same way you'd review any other production quality metric, retiring or re-tuning alerts that have stopped being actionable.
The cheapest observability investment is the one made before the incident that would have justified it.MNB DevOps Practice
Key actions
- Design instrumentation around specific likely failure scenarios
- Connect logs, metrics and traces with consistent request IDs
- Review alert precision on a regular cadence
- Retire alerts the team has learned to ignore