Unit
- MapReduce limitations: MapReduce, introduced by Google in 2004, provided a programming model for processing large datasets on clusters. However, it writes intermediate results to disk between the map and reduce phases, causing high latency for iterative algorithms and interactive queries.
- Spark's innovation: Spark emerged from UC Berkeley AMPLab in 2009 as a response to MapReduce's shortcomings. It introduced in-memory cluster computing, where data is cached in memory across operations, reducing disk I/O and accelerating iterative workloads like machine learning and graph algorithms.
- Resilient Distributed Datasets (RDDs): Spark's core abstraction is the RDD, a fault-tolerant collection of elements partitioned across the cluster. RDDs can be created from Hadoop InputFormats or by transforming other RDDs. Their lineage graph enables recomputation of lost partitions without replication.
- Speed advantage: By keeping data in memory and leveraging directed acyclic graph (DAG) execution, Spark achieves 10-100x faster performance than MapReduce for many workloads. The DAG scheduler optimizes the execution plan, minimizing shuffle operations.
- Unified engine: Unlike earlier systems that required separate tools for batch (MapReduce), streaming (Storm), and SQL (Hive), Spark provides a single runtime for all these workloads using the same core API. This reduces operational complexity and code maintenance.
- Adoption and ecosystem: Spark became a top-level Apache project in 2014 and is now used by thousands of organizations. Its ecosystem includes Spark SQL, Structured Streaming, MLlib, and GraphX, making it versatile for data engineering, data science, and analytics.
- Transition to DataFrames: RDDs are low-level. Spark later introduced DataFrames, a schema-aware abstraction inspired by data frames in R and Python, enabling catalyst optimizer and SQL queries. This improved performance and developer productivity.
- Modern evolution: With Apache Spark 3.x, features like adaptive query execution, dynamic partition pruning, and enhanced ANSI SQL support continue to push the boundaries of distributed data processing, making it suitable for modern data lakes and lakehouse architectures.
- Lazy evaluation: Spark transformations are lazy, meaning they do not compute results immediately. Instead, they build a lineage graph. When an action like count or collect is called, Spark optimizes the entire computation plan. This minimizes data movement and allows the Catalyst optimizer to apply rule based optimizations for efficient execution.
- Cluster deployment options: Spark can run on various cluster managers including Apache Hadoop YARN, Apache Mesos, and Kubernetes. It also supports standalone mode for simple setups. This flexibility allows organizations to integrate Spark into existing infrastructure while benefiting from resource scheduling and dynamic allocation across multiple nodes.
- Shared variables: Spark provides two types of shared variables, broadcast variables and accumulators. Broadcast variables cache a read only value on each node, avoiding data shuffling for large lookup tables. Accumulators are write only variables used for counters and sums, ensuring fault tolerant aggregation across tasks.
Code Example
Python
from pyspark import SparkContext
sc = SparkContext('local', 'MapReduce Comparison')
data = sc.parallelize(range(1, 10000000))
# Simulating MapReduce: each number squared, filter even
squares = data.map(lambda x: x*x)
even_squares = squares.filter(lambda x: x % 2 == 0)
count = even_squares.count()
print('Count: {}'.format(count))
sc.stop()This code creates an RDD of numbers, applies map and filter transformations, and triggers an action (count). Spark's lazy evaluation defers computation until the action, building a DAG. The result shows how Spark processes data in memory without intermediate disk writes.
Quick Revision
- MapReduce writes intermediate data to disk, causing high latency.
- Spark improves performance through in-memory computation and DAG execution.
- RDDs are fault-tolerant collections that can be created and transformed.
- Spark evolves with DataFrames for better optimization and usability.