Fundamentals of HBase

I started reading about Hbase pretty recently and I’ll be writing down the way I read and understood it. There might be certain things out of order but I’ll update those accordingly.

So what’s so special about Hbase? Well it’s a distributed db, first that I encountered, which makes me wonder how does it do what other databases do? Or if it’s better, how so?

Apparently from the Apache Hbase docs we have this:

HBase isn’t suitable for every problem.

First, make sure you have enough data. If you have hundreds of millions or billions of rows, then HBase is a good candidate. If you only have a few thousand/million rows, then using a traditional RDBMS might be a better choice due to the fact that all of your data might wind up on a single node (or two) and the rest of the cluster may be sitting idle.

Second, make sure you can live without all the extra features that an RDBMS provides (e.g., typed columns, secondary indexes, transactions, advanced query languages, etc.) An application built against an RDBMS cannot be “ported” to HBase by simply changing a JDBC driver, for example. Consider moving from an RDBMS to HBase as a complete redesign as opposed to a port.

Third, make sure you have enough hardware. Even HDFS doesn’t do well with anything less than 5 DataNodes (due to things such as HDFS block replication which has a default of 3), plus a NameNode.

HBase can run quite well stand-alone on a laptop — but this should be considered a development configuration only.

This means that if we can estimate our data to not go beyond a certain threshold we can still go ahead and work with the traditional single node RDBMS.

A question might come up here. RDBMS can also scale and we can partition the data as well. What is HBase solving?

According to my understanding, we’ll first have to setup a ProxySql that would help route requests to different SQL servers based on routes and rules. But when the loads grows even further we’ll have to introduce multiple ProxySql servers and then setup a load balancer in front of them. Too many custom setup and it’s good for multiple use cases. However, Hbase gives everything out of the box and hence the cost of setup is high. So estimate the volume of data before deciding which DB to go with.

Now let’s talk about the storage architecture of Hbase.

• Rows: Each row has a unique row key that acts as the primary key.
• Column Families: Groups of related columns that are stored together. They must be defined when creating a table.
• Column Qualifiers (Columns): Individual data elements within a column family, identified by a qualifier or column name.
• Cells: The intersection of a row and column, containing a value and a timestamp.
• Timestamps: Each value in HBase is versioned with a timestamp.

Cell is the smallest unit of data. It’s identified by the combination of

• Rowkey
• Column Family
• Column Qualifier
• Timestamp

Row is formed by all the Cells that share the same Rowkey.

There’s no type enforcement on row keys. We could have one rowkey as int and another as string but this is discouraged since they are sorted lexicographically and would cause problems while scanning. So we should always model our rowkeys accordingly.

In Hbase, Column is comprised of two parts

• Column family
• Column qualifier - They’re typically denoted as columnFamily:columnQualifier (cf:cq).

Now what’s the difference between a column family and a column qualifier.

Column Family:

• It is generally a high level segregation of columns.
• The biggest distinction for a column family is the physical storage separation. Meaning the data for the same column family are physically stored together on HDFS. We can leverage this by making sure that we keep related data (relational data in SQL terms) together for easy retrieval.
• Column families must be predefined when creating the Hbase table. We cannot add new column families on the fly without altering the schema.
• Certain storage properties such as compression, ttl, number of versions(can be used for storing historic data as well), caching etc can be defined on the column family level.
• Ideally, we should try to keep the number of column families small.

Column Qualifier:

• They are schema less, they can be added on the fly. We can have multiple column qualifiers to the count of millions and even then it would be performant.
• No type enforcement, meaning we can store int for one row and string for another. The interpretation is supposed to be done by the application.

Now we span out to see the bigger picture here. The bigger moving components of HBase are as below:

• Region: Portions of tables split horizontally by row key range, the basic unit of scalability.
• RegionServer: Physical server that hosts and serves regions to clients.
• HMaster: Coordinates the cluster and performs administrative operations.
• ZooKeeper: Manages the distributed coordination service for HBase.
• HDFS: The underlying distributed file system where HBase stores its data.

Region Multiple sorted rows create a Region. These are what can be called horizontal partitioning of a table. As discussed earlier, since each column family data is kept together physically this means that for each column family we have at least 1 hfile created. Once the size of that hfile increases beyond a certain limit namely the configuration hbase.hregion.max.filesize which defaults to 10GB it triggers a split. It creates daughter regions that references to the same hfile till the time anyone of them get a new write and memstore flushes the data to the new hfile.

RegionServer Now a RegionServer is a collection of Regions. They temporarily store writes in memory (MemStore) before persistently writing then as HFiles to HDFS.

Q: How does it know which region to get the data from?

So region server is actually responsible to deal with the requests, it receives a requests, figures out which region to get the data from and then proceeds further. RegionServer uses an internal in memory map to figure out which region to talk to, in order to fetch the data. So in general RegionServer are responsible for handling the requests.

HMaster This servers as the master server in an Hbase cluster. It is responsible for

• Schema Operations: Creating, deleting, and altering tables
• Region Assignment: Deciding which RegionServer should host which regions
• Load Balancing: Moving regions between RegionServers to distribute load evenly
• Failover Management: Detecting failed RegionServers and reassigning their regions
• Metadata Management: Maintaining table metadata and region information

There could be multiple HMasters for high availability but only one of them will be active at any point in time.

ZooKeeper This is crucial for:

• Cluster coordination: It keeps track of available regionservers and active Hmaster.
• Leader Election: If we’re hosting more than one Hmaster, and/or if any Hmaster goes down, it helps in electing an active Hmaster.
• Failure detection: In case a regionserver goes down, ZK notifies the Hmaster to do the needful.

This was the absolute basics of the internal components of Hbase. Didn’t go much in deep but will cover the read, writes along with compaction and best data modelling practices in the next article.

Continuation: https://medium.com/@shahidmsj27/more-about-hbase-218b013cea0e