Mastering Hadoop: A Deep Dive into Distributed Data Processing

Overview

Hadoop is an open-source framework for the parallel distributed storage and processing of large datasets. Developed by the Apache Software Foundation, it provides a scalable, fault-tolerant architecture that enables organizations to handle Big Data efficiently. At its core, Hadoop follows a MapReduce programming model and uses HDFS (Hadoop Distributed File System) for storage, ensuring seamless data processing across multiple nodes.

Business Scenarios Where Hadoop is Used

Hadoop is widely used in industries dealing with massive data volumes. Some everyday use cases include:

1. Telecom Industry — Call Data Records (CDR) processing, subscriber analytics, Billing systems, and fraud detection.
2. Banking & Finance — Risk modeling, real-time transaction monitoring, and fraud detection.
3. E-commerce & Retail — Recommendation engines, inventory management, and user behavior analysis.

4. Social Media & Web Analytics — Log processing, sentiment analysis, and targeted advertising.

Key Features of Hadoop

• Scalability →Handles petabytes of data across multiple worker nodes.
• Fault Tolerance →Data replication across multiple nodes ensures reliability even if nodes fail.
• Cost-Effective →Uses commodity hardware, reducing infrastructure costs.
• High Throughput →Supports parallel processing with MapReduce processing.
• Flexibility →Can process structured, semi-structured, and unstructured data.

Hadoop Architecture

Hadoop follows a Master-Slave architecture containing:

Hadoop | Hadoop Architecture | HDFS

Main Components of Hadoop

1-HDFS (Hadoop Distributed File System)

Stores large files across multiple machines. Composed of NameNode (Master) and DataNodes (Slaves OR Worker Nodes).

2-MapReduce Framework

A programming model that processes data in two steps: Map (splitting & filtering) and Reduce (aggregating results).

3-YARN (Yet Another Resource Negotiator)

Manages resource allocation and job scheduling across the cluster.

Architecture Components and their roles:

Below is a visual representation of the Hadoop architecture:

Hadoop follows a Master-Slave architecture, consisting of Master Nodes and Worker Nodes.

Hadoop | Hadoop Architecture | HDFS | MapReduce | Yarn

1-Master Node Components

The Master node is responsible for coordinating the cluster, managing storage (HDFS), and scheduling jobs (YARN).

NameNode & Secondary NameNode

NameNode → Manages metadata and keeps track of file system hierarchy. It does not store actual data but maintains references to where files are stored.

Secondary NameNode → Periodically saves metadata snapshots to prevent loss in case of NameNode failure.

YARN (Resource Manager)

Manages cluster resources and schedules application execution. Ensures optimal utilization by allocating CPU, memory, and processing power to different jobs.

MapReduce Server (Job History Server)

Stores logs and execution history of completed MapReduce jobs. Helps in debugging and analyzing past jobs

2-Worker Node Components

Worker nodes perform the actual data storage and processing tasks.

DataNode

Stores actual data blocks. Continuously communicates with the NameNode to report block locations and status.

Node Manager (YARN)

Manages resources on each worker node. Communicate with the YARN Resource Manager for job execution.

Application Master

Manages the lifecycle of applications running on Hadoop. Requests resources from YARN and monitors execution.

MapReduce Task Containers

Execute the Map and Reduce tasks within allocated containers. Tasks are dynamically assigned based on available resources.

How These Components Work Together?

1. Data Ingestion: Files are split into blocks and distributed across DataNodes.
2. Job Execution: YARN assigns resources, and MapReduce processes data in parallel.
3. Fault Tolerance: If a node fails, Hadoop reassigns tasks and retrieves data from replicated copies.

Let’s explore every aspect of each module to see how things are working.

HDFS Internals:

1. Block Storage and replication:

→ Files in HDFS are split into blocks (default 128MB/256MB) and distributed across DataNodes.

→ Each block is replicated (default factor: 3) to ensure data redundancy.

→ The NameNode keeps track of block locations, while DataNodes store actual data.

2. Write Pipeline in HDFS

To understand the pipeline in Hadoop first, you need to understand the rack awareness concept of Hadoop.

**Note*: What is rack awareness in Hadoop?

Rack awareness is a feature where the system is aware of the physical network topology, allowing it to distribute data and tasks intelligently across racks to improve fault tolerance, network performance, and overall cluster efficiency

When a file is written:

→ The client contacts the NameNode for block allocation.

→ The NameNode selects three DataNodes (based on rack awareness).

→ Data flows from the client → first DataNode → second DataNode → third DataNode in a pipeline.

→ Once all DataNodes acknowledge the write, the block is considered successfully stored.

3. Read Process in HDFS

→ The client queries the NameNode to get block locations.

→ Data is fetched from the nearest available DataNode (data locality principle) to reduce network overhead.

YARN:

Yet another resource negotiator is the Hadoop resource management layer part, used for job scheduling.

1. Components of YARN:

→ Resource Manager (RM) →Allocates cluster resources and schedules applications.

→ Node Manager (NM) →Runs on each worker node, and manages resources locally.

→ Application Master (AM) →Manages the execution of a specific application.

2. How Resource Allocation Works?

→ The Application Master requests resources from the Resource Manager.

→ The Scheduler in the resource manager assigns CPU & memory based on availability and priority.

→ Node Managers launch the allocated containers to execute the tasks.

3. Scheduling Policies in YARN

→ FIFO Scheduler — Jobs are executed in First-In-First-Out order.

→ Capacity Scheduler — Ensures fair resource allocation among multiple users.

→ Fair Scheduler — Dynamically adjusts resources for fairness.

How Fault Tolerance Works in Hadoop?

Hadoop is designed to handle node failures without losing data or job progress.

1. HDFS Fault Tolerance

→ Data Replication: If a DataNode fails, other replicas provide the missing data.

→ Automatic Re-replication: The NameNode detects missing blocks and re-replicates them.

2. MapReduce Fault Tolerance

→ Task Retry: Failed tasks are automatically restarted on another node.

→ Speculative Execution: If a task is running slow, a duplicate is launched on another node.

3. Yarn Fault Tolerance

→ Application Master Recovery: If the Application Master fails, it restarts automatically.

→ Resource Manager HA: A standby RM takes over if the active one crashes.

How do Checkpoints help in fault tolerance?

**Note*: what are checkpoints in Hadoop?

Checkpoints help in preventing metadata loss and speeding up NameNode recovery. Role of secondary NameNode

The Secondary NameNode takes periodic snapshots (checkpoints) of the NameNode‚ metadata.

These snapshots are stored as FsImage (file system image) and edit logs.

How its work?

1. The Secondary NameNode merges the latest FsImage with the edit logs.
2. It saves the cleaned snapshot and provides it to the NameNode during recovery.
3. This prevents loss of metadata in case of NameNode failure.

Hadoop Data Flow & Internal Communication Protocols:

Hadoop operates as a distributed system where clients, NameNodes, DataNodes, and YARN components interact using various protocols for authentication, data transfer, and resource management. Below is a detailed step-by-step breakdown of how Hadoop handles data read, write, and internal communication(RPC) with a focus on protocols used at each stage.

**RPC*= Remote procedure call acts like a low-level transport protocol for carrying data packets between communicating programs. It does this using a client-server model.

Hadoop Communication protocols | RPC | HDFS | Hadoop

Writing/Reading Data to HDFS (Hadoop RPC + HDFS Data Transfer Protocol)

Step 1: Client Requests File Upload

The client contacts NameNode via Hadoop RPC to create a new file. NameNode checks Does the file already exist? Does the client have write permissions?

Protocol Used = RPC

Step 2: Data Transfer to DataNodes

The client splits the file into blocks (default: 128MB each and also this is configurable). HDFS Data Transfer Protocol establishes a pipeline to transfer data.

Client → DataNode1 → DataNode2 → DataNode3

Each DataNode forwards the block to the next node to maintain replication (default: 3 copies).

Protocols Used: HDFS Data Transfer Protocol, TCP/IP

Step 3: DataNode Acknowledgement

Each DataNode sends an ACK to the client via Hadoop RPC confirming block storage. The client notifies the NameNode that the writing is complete.

Protocol Used = RPC

Why we are studying protocols?

Protocols are the silent architects of distributed systems — understanding them is the key to unlocking how data moves, transforms, and scales efficiently.

Internal Node Communication (IPC + ZooKeeper + Netty)

→ Heartbeats & Block Reports (IPC + RPC)

DataNodes send heartbeats to the NameNode every 3 seconds via Hadoop IPC (Inter-Process Communication). Every 6 hours, DataNodes sends a Block Report to inform the NameNode about stored data.

Protocols Used: IPC (for intra-cluster communication), Hadoop RPC

Benefit: Keeps NameNode updated on cluster health.

→ Leader Election & Cluster Coordination (ZooKeeper Protocol)

If the NameNode fails, Hadoop automatically elects a new leader using Apache ZooKeeper. ZooKeeper ensures that only one active NameNode manages metadata at a time.

Protocols Used: ZooKeeper Consensus Protocol

Benefit: Ensures high availability and prevents metadata conflicts.

→ YARN Resource Management (Netty + RPC)

Clients submit jobs to YARN’s Resource Manager via RPC. ResourceManager communicates with NodeManagers using Netty (an asynchronous RPC framework).

NodeManagers allocate containers for running tasks.

Protocols Used: Netty (for async messaging), RPC (for job scheduling)

Fault Tolerance & Checkpoints (ZooKeeper + Secondary NameNode + Replication Protocols)

Handling NameNode Failure

The Secondary NameNode periodically stores checkpoints (metadata snapshots) of the active NameNode. If the primary NameNode fails, it can be restored from the latest checkpoint.

Protocols Used: ZooKeeper for failover, HDFS Checkpoint Protocol

Handling DataNode Failure

If a DataNode goes down, the NameNode detects the failure due to missing heartbeats. NameNode re-replicates lost data to other DataNodes using HDFS Replication Protocol.

Protocols Used: HDFS Replication Protocol, TCP/IP

Conclusion

Hadoop has revolutionized the way we handle large-scale data processing by providing a distributed, scalable, and fault-tolerant framework. With its core components — HDFS for storage, YARN for resource management, and MapReduce for processing — it ensures efficient data handling across clusters. The seamless communication between nodes, enabled by Hadoop’s robust RPC mechanisms and ZooKeeper coordination, makes it a powerful solution for big data challenges.

Understanding Hadoop’s internals is crucial for anyone working with large datasets, as it helps in optimizing performance, ensuring reliability, and leveraging its full potential. Whether in telecom, finance, or AI-driven analytics, Hadoop remains a key player in the world of big data.

Mastering Hadoop isn’t just about using it — it’s about understanding how it works under the hood. Keep exploring, keep questioning, and let the data guide you forward!