Unit
- Definition: Infrastructure monitoring is the practice of collecting, analyzing, and visualizing data from hardware, software, and network components to ensure system health. It provides real-time visibility into performance, availability, and resource utilization, enabling teams to detect and respond to issues before they affect users.
- Key Objectives: The primary goals include maximizing uptime, optimizing resource usage, identifying bottlenecks, and supporting capacity planning. Monitoring helps teams understand normal behavior patterns and quickly identify deviations that indicate problems.
- Monitoring vs. Observability: Monitoring is the act of measuring known metrics against predefined thresholds, while observability explores unknown unknowns through high-cardinality data and traces. Both are complementary: monitoring handles alerts, observability aids root cause analysis.
- The Four Golden Signals: According to Google SRE practices, the four key metrics are latency (time to service requests), traffic (demand measured in requests per second), errors (rate of failed requests), and saturation (resource utilization). These signals provide a comprehensive view of system health.
- Types of Monitoring: System monitoring covers CPU, memory, disk, and network. Application monitoring tracks response times, error rates, and custom business metrics. Network monitoring analyzes bandwidth, packet loss, and connectivity. Each type feeds into a unified observability strategy.
- Push vs. Pull Models: In pull-based monitoring, the monitoring server scrapes metrics from targets at regular intervals (e.g., Prometheus). In push-based, agents send metrics to a central collector (e.g., StatsD, Graphite). Choice depends on architecture scale and network constraints.
- Tool Categories: Open-source options include Prometheus, Grafana, Nagios, and Zabbix. Cloud-managed services include AWS CloudWatch, Azure Monitor, and Google Cloud Operations. Many organizations use a combination to cover different requirements.
- Challenges and Pitfalls: Common issues include alert fatigue from false positives, lack of proper metric cardinality management, and missing context during incident response. Investing in alert tuning and proper dashboard design mitigates these risks.
- Metric Types and Cardinality: Common metric types include counters (cumulative values), gauges (point-in-time snapshots), and histograms (distribution of values). High cardinality occurs when metrics have many unique label combinations, straining storage and query performance. Implement aggregation and label whitelisting to manage cardinality effectively without losing observability.
- Alerting Strategies: Effective alerts should be symptom based rather than cause based, reducing noise from false alarms. Use static thresholds for known patterns, anomaly detection for dynamic baselines, and multiwindow evaluation to confirm sustained issues. Regular review and tuning of alert rules prevent alert fatigue and ensure critical signals are not missed.
- Integration with Incident Response: Monitoring dashboards and alerts should link directly to runbooks that guide responders through diagnosis. Automated triggers can create tickets, notify on-call teams, and initiate remediation workflows. Post-incident audits use monitoring data to identify gaps in coverage and improve alerting logic.
Code Example
Python
from prometheus_client import start_http_server, Counter
import random, time
requests_total = Counter('http_requests_total', 'Total HTTP Requests')
def process_request():
requests_total.inc()
time.sleep(random.random() * 0.1)
if __name__ == '__main__':
start_http_server(8000)
while True:
process_request()This script starts a Prometheus metric server that exposes a counter of processed requests. The start_http_server function launches an HTTP endpoint on port 8000 that Prometheus can scrape.
Quick Revision
- Infrastructure monitoring collects data from all layers of the stack to ensure system health.
- The four golden signals are latency, traffic, errors, and saturation.
- Monitoring differs from observability in scope and depth.
- Push and pull models have different tradeoffs for scalability and simplicity.
- Tools like Prometheus and Grafana help implement effective monitoring pipelines.