Unit
- Event streams: A stream is an ordered sequence of events. Each event represents a fact about something that happened, such as a user click, a sensor reading, or a transaction. Events are immutable, time-stamped, and produced continuously.
- Immutability: Events in a stream are append-only; they cannot be modified after creation. This property simplifies reasoning about data history and enables reproducible reprocessing from any point in time.
- Continuous flow: Unlike static datasets, streams are unbounded. Data arrives continuously and with varying velocity, requiring systems that never stop consuming and processing.
- Low latency: Streaming systems aim for sub-second processing from event arrival to output. This demand drives architectural choices like in-memory computation and incremental processing.
- Event time vs. processing time: Event time is the timestamp when the event actually occurred, embedded in the data. Processing time is when the system observes it, often different due to delays or batching.
- Use case - real-time analytics: Companies monitor dashboards for live metrics like active users, revenue per minute, or error rates. Streaming allows immediate action when thresholds are crossed.
- Use case - fraud detection: Financial transactions are scored in real-time against historical patterns. Suspicious activity triggers alerts or blocks within milliseconds of the transaction.
- Use case - IoT sensor monitoring: Millions of devices send telemetry data such as temperature, vibration, or location. Streaming pipelines aggregate, filter, and trigger alerts for anomalies.
- Windowing: Streaming systems group events into finite windows for aggregation. Common window types include tumbling, sliding, and session windows. Windows are defined by time duration or event count. They enable calculations like sum, average, or count over recent data, updating results as each window completes or advances.
- Watermarks: A watermark is a metric that tracks the progress of event time. It indicates that all events with a timestamp less than the watermark have been observed. Systems use watermarks to trigger window computations and handle late arriving data, discarding or reprocessing events beyond a configurable threshold.
- Exactly once semantics: This guarantee ensures that each event is processed exactly once, even in the face of failures. It is achieved through idempotent producers, transactional writes, and offset management. Exactly once prevents duplicates and data loss, critical for financial and audit systems.
System Architecture
Pattern: Event-Driven Architecture using Apache Kafka
Components:
- Producers: Applications or devices that publish events to Kafka topics. Each event contains a key, value, and timestamp.
- Brokers: Kafka servers that store and serve events. Brokers replicate partitions across the cluster for fault tolerance.
- Topics and partitions: A topic is a logical channel. Each topic is split into partitions for parallelism. Partitions are ordered, immutable sequences of records.
- Consumers: Applications that subscribe to topics and process events. Consumers track their offset to ensure exactly-once processing.
- Schema Registry: External service that stores and validates event schemas (Avro, Protobuf). It enforces backward compatibility.
Code Example
Python
from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='localhost:9092')
producer.send('user-clicks', key=b'user123', value=b'click_event')
producer.flush()This Python snippet creates a Kafka producer that sends a click event to the user-clicks topic. The producer uses a key for partition routing and flushes to ensure delivery.
Quick Revision
- Streams are immutable, ordered, continuous sequences of events.
- Low latency processing is crucial for real-time applications like fraud detection.
- Event time differs from processing time; systems must handle both.
- Apache Kafka is a popular broker providing durability and scalability.