HDFS Architecture Explained: A Beginner-Friendly Guide to Hadoop Storage

Introduction

In today’s digital world ,companies generate massive amount of data every second. Traditional storage system fails to handle massive amount of data. This is where Hadoop Distributed File system (HDFS) comes in.

HDFS is a core component of the Hadoop ecosystem, designed to store and process large datasets reliably and efficiently.

In this blog, you’ll learn:

• what HDFS is
• how HDFS architecture works
• key components and their roles
• Read & write process
• advantage and limitations

what is HDFS?

HDFS (Hadoop Distributed File System) is a distributed storage system that splits a large file into smaller blocks and stores them across multiple machines.

key Features:

• fault tolerant
• scalable
• high availability
• designed for large files

key components of HDFS Architecture

HDFS mainly consist of three components:

1. Name Node (Master Node)

The name node is the brain of HDFS

• stores meta data (file names, locations, permissions)
• manages file system namespace
• keep track of block location

2.Data Node (Worker Nodes)

Data Nodes are responsible for storing actual data.

• store data blocks
• perform read/write operations
• send heartbeat signal to Name Node

3.Secondary Name Node

despite its name, it is Not a backup Name Node

• performs checkpointing
• Merges metadata logs
• Helps reduce Name Node load

HDFS Architecture overview

HDFS follows a distributed architecture where:

• Files are split into blocks
• Blocks are stored across multiple Data Node
• Meta Data manages by Name Node

How Data is stored in HDFS

step-by-step process

1. A file is divided into blocks (default size:128MB)
2. Each block is stored in different Data Nodes
3. Each block is replicated (default replication factor=3)
4. Name Node stores meta data (block locations)

Example:

File size:500MB

blocks created:4

Each block is stored in multiple nodes

HDFS Write operations

How data is written:

1. client sends request to name node
2. Name node returns data node locations
3. Data is written to data nodes
4. Replications happen automatically

data is written in a pipeline manner.

HDFS Read operations:

How data is read:

1. client request Name node
2. Name node provides block locations
3. client reads data from nearest data nodes

This improves performance using data locality.

Fault Tolerance in HDFS

HDFS is designed to handle failures efficiently

• Replication
• Heart Beat signals
• Automatic recovery
• Rack awareness

Advantages of HDFS

• Highly scalable
• Fault tolerance
• Cost effective
• suitable for big data processing

Limitations of HDFS

• Not suitable for small files
• High latency
• Not ideal for real-time processing