Hello All!

Hadoop Framework Components

Hello All! This blog gives you insights into the components of Hadoop framework(a big data technology for storage and processing data). Hadoop Framework mainly consists of 🧿HDFS(Hadoop Distributed File System)- distributed file storage for handling big data

🧿MR(Map Reduce)- for parallel data processing, primarily uses Java

MapReduce abstractions include:

📽️ Hive - a query engine thats an alternate for Java but still using MapReduce wherein the programmers write code in SQL, but in backend its converted into Java. Performance similar to MR.

Most extensively used in big data. 📽️ Pig - programmers uses pig latin scripting language to write code, but in backend converts to Java

📽️Scoop - When importing or exporting data, each time code has to be written in MR.Hence to solve this problem Scoop is being used for importing data from RDBMS to Hadoop and exporting from Hadoop to RDBMS. It’s also used for data pipeline.

📽️ Oozi - its like an xml file, used as a scheduler(to automatically run the program at a specific time)

Other components of Hadoop Framework include 🧿Hbase- Hadoop NoSQL database

🧿Mahout - a data science component for Hadoop

🧿Flume- Its similar to Scoop, retrieves data in real time from rdbms ,web servers etc. to only Hadoop,back Hadoop to multiple targets not possible.

Summarizing it into different layers: 💽 Storage - HDFS,HBase 🖨️ Processing- MR,Hive,Pig 🗜️ Data pipeline- Scoop ,Flume ⏰ Scheduler - Oozi 📈 Data science - Mahout

Will continue posting related to data engineering!