Unit
- Monitoring: Monitoring is the process of collecting, analyzing, and using predefined metrics and logs to track known failure modes and ensure system availability. It provides answers to known questions about system health based on thresholds and patterns.
- Observability: Observability is a property of a system that allows teams to infer its internal state from external outputs without needing to deploy new instrumentation. It enables exploration of unknown unknowns through rich telemetry data.
- Difference in scope: Monitoring focuses on known failure modes and predefined dashboards, while observability emphasizes the ability to debug novel issues by examining high-dimensional data like distributed traces and structured logs.
- Why both matter: In complex distributed systems, monitoring catches common problems quickly, but observability is essential when new failure modes emerge. Teams should invest in both to maintain reliability.
- Key components: A monitoring system includes data collection agents, storage backends, alerting engines, and visualization tools. Observability adds tracing infrastructure and log aggregation with search capabilities.
- Common pitfalls: Relying solely on monitoring can lead to blind spots for rare or unprecedented failures. Conversely, over-instrumenting for observability without actionable alerts causes data overload.
- Organizational impact: Teams with strong monitoring and observability practices resolve incidents faster, reduce downtime, and build confidence in their systems. It also supports data-driven decisions for capacity planning and performance optimization.
- Getting started: Begin with monitoring critical user journeys and infrastructure metrics, then gradually add tracing and structured logging as system complexity grows. Iterate based on incident postmortems.
- Alerting fundamentals: Alerting transforms raw monitoring data into actionable notifications. Effective alerts are based on clear thresholds, maintenance windows, and escalation policies. Avoid alert fatigue by tuning alert sensitivity and consolidating related signals. Each alert should have a runbook that guides the responder through diagnosis and resolution steps.
- Service level objectives SLOs: SLOs define target reliability levels for services and align monitoring with user experience. By measuring error budgets, teams can prioritize feature development versus reliability work. SLO based alerting uses burn rate or exhaustion rate to trigger alerts only when the error budget is being consumed too quickly, reducing noise.
- On call best practices: On call rotations require clear handoff procedures, a dedicated escalation path, and access to runbooks and dashboards. Teams should implement a blameless postmortem culture to learn from incidents. Regular practice with chaos engineering or game days helps on call engineers stay prepared for real incidents.
- Integration with incident management: Monitoring and alerting systems should integrate with incident management platforms like PagerDuty or Opsgenie. These tools provide automatic ticket creation, step down escalation, and context rich notifications. Integration ensures that the right people are notified based on schedule, severity, and the service affected.
Code Example
Python
import requests
import sys
url = 'http://localhost:8080/health'
try:
response = requests.get(url, timeout=5)
if response.status_code == 200:
sys.exit(0)
else:
sys.exit(1)
except requests.exceptions.RequestException:
sys.exit(1)This script performs a basic HTTP health check. A zero exit code indicates success, non-zero indicates failure. It can be used in cron jobs or monitoring agents to verify service availability.
Quick Revision
- Monitoring tracks known failure modes; observability explores unknown issues.
- Both are needed for comprehensive system reliability.
- Start with monitoring and evolve toward observability as systems grow.
- Use health checks as a minimal monitoring starting point.