Unit
- HDFS: The Hadoop Distributed File System provides fault-tolerant storage across a cluster. Data is split into blocks (default 128 MB) and replicated across multiple data nodes. The NameNode manages metadata and coordinates access, while DataNodes store actual data blocks. This architecture ensures reliability even when individual nodes fail.
- YARN: Yet Another Resource Negotiator is the cluster resource management layer. It separates resource allocation from job scheduling. The ResourceManager allocates cluster resources to applications, while per-node NodeManagers manage containers. This allows multiple processing engines (MapReduce, Spark) to share the same cluster.
- MapReduce: A programming model for parallel data processing. Map tasks process input key-value pairs and produce intermediate results, which are shuffled and sorted before Reduce tasks aggregate them. Although newer engines like Spark are faster, MapReduce remains foundational to understanding Hadoop's processing paradigm.
- Common Utilities: A set of libraries and utilities that support other Hadoop modules. These include filesystem abstractions, remote procedure call (RPC) mechanisms, and serialization formats. They enable consistent operation across different Hadoop components and simplify development of distributed applications.
- Ecosystem Projects: Hadoop is often used with complementary tools like Hive (SQL-like queries), Pig (data flow scripting), HBase (NoSQL database), and Spark (in-memory processing). Administrators must understand how these interact with core Hadoop services to manage dependencies and resource contention.
- Cluster Roles: A typical Hadoop cluster has master nodes (running NameNode, ResourceManager) and worker nodes (DataNode, NodeManager). Master nodes handle coordination and metadata, while worker nodes store data and execute computations. Balancing these roles is critical for performance and resilience.
- Data Locality: Hadoop's design moves computation to data rather than data to computation. When a MapReduce job runs, YARN attempts to schedule tasks on nodes where the data blocks reside. This minimizes network traffic and improves throughput, especially for large datasets.
- Versioning and Compatibility: Hadoop has evolved through multiple major releases (1.x, 2.x, 3.x). Administrators should be aware of API changes, deprecations, and compatibility between ecosystem components. Using a consistent distribution (e.g., Apache Hadoop, Cloudera, Hortonworks) simplifies management.
- Security Integration: Hadoop clusters require robust authentication and authorization mechanisms. Kerberos is the primary authentication protocol, validating users and services via tickets. Apache Ranger or Sentry can enforce fine grained access controls on HDFS directories and Hive tables. Administrators must configure keytabs, manage principals, and ensure secure communication between nodes.
- High Availability: NameNode high availability eliminates single points of failure. Two NameNodes run in active standby mode, sharing edit logs via a Quorum Journal Manager. If the active NameNode fails, the standby automatically takes over. This configuration requires careful setup of ZooKeeper, JournalNodes, and shared storage to ensure seamless failover without data loss.
- Rack Awareness: Hadoop uses rack awareness to optimize data placement and network traffic. Administrators define a network topology script that maps each node to a rack. The NameNode then replicates data blocks across multiple racks to ensure tolerance against rack level failures. This reduces bandwidth usage and improves job performance by favoring local rack reads.
Code Example
Shell
hdfs dfs -ls /user/data
hdfs dfs -put localfile.txt /user/data/
hdfs dfs -cat /user/data/localfile.txt
yarn application -list
mapred job -list allThese commands interact with HDFS and YARN. hdfs dfs commands perform file operations. yarn application -list shows running applications. mapred job -list displays MapReduce jobs. Understanding these basic operations is essential for daily administration tasks.
Quick Revision
- HDFS provides distributed, replicated storage with NameNode metadata management.
- YARN separates resource management from job scheduling, supporting multiple frameworks.
- MapReduce is the original processing model using map and reduce phases.
- Ecosystem tools extend Hadoop's capabilities for SQL, scripting, and real-time processing.
- Master nodes run coordination services; worker nodes store data and run tasks.
- Data locality places computation near data to reduce network overhead.
- Version compatibility between Hadoop and ecosystem components must be maintained.