Everything about Elastic search (ES Cloud & Health) : Part 2

This is in continuation in the series of Elastic search ( Part 1 )

1. Part 2 covers the importance of health checks either by monitoring dashboards in ES Cloud / Kibana/ API.
2. The steps to resolve unassigned shards in elastic search &
3. The offerings on Elastic Search Service Cloud.

Elastic Search Service Cloud

The ES cloud allows to Manage all deployments from a single console with simplified, automated options using their dashboard and API’s. we can spin up a fully loaded deployment with a host of Cloud providers like AWS, GCP, Azure et al. Some of the key features on the Elastic Cloud Service ( I hosted on AWS )

Elastic Search Cloud support hosting on AWS, GCP, Azure

Some of the key features include :

1. Metrics ( over all stack metrics in terms of CPU, memory, disk among others)
2. Application Performance Monitoring (APM) collects in-depth performance metrics and errors from inside the application.
3. Force update or restart ( button click away : Ex: restarting just Kibana or restarting the entire ES cluster : All nodes and it also has options to restart with and w/o downtime )
4. API Console ( to query ES APIs )
5. Logs ( Entire ES logs can be viewed in one place )
6. Snapshot ( restoring is pretty simple and button click away )
7. One-click upgrades; which mean getting the latest version of Elasticsearch and Kibana with no downtime.
8. Health of the Instances in the Dashboard We could see the over all Elastic Search Cloud deployment instance config and health status right from the default cloud dash.
9. We can generate API keys for the account and leverage for usage in making RESTful requests via console
10. Deployment Edit allows Simple Management for the users to enable the entire ES stack infra with just few button click for different features to scale up both vertically and horizontally. Indeed we can click, scale with a slider bar.
11. Security ( To ensure additional measures like traffic filters that could be enabled to protect the cloud )
12. Activity ( Chronological logging of entire activity on the ES cloud )
13. Elastic Search plugins ( Just create an extension and Drag and drop as plugin)

Sample of ES Cloud stack deployed on AWS ||| Metrics of the ES Cluster Cloud ||| Snapshots enable easy restore & easy backup

Photo by Tim Gouw on Unsplash

Elastic Search Cluster Health Status using kibana (ES Stack Monitoring)

We could easily setup with a button click to enable the entire ES monitoring in Kibana. Its easy for on-premise ES to manage ES cluster and stack health. We can visually see the over all health of entire cluster, indices, Nodes, Kibana et al

1. Green means everything is healthy
2. The common cause of Yellow status mean not having enough nodes in cluster for primary or replica shard.
3. Red status means at any critical issue with over all cluster for example on the shard level, Red means at least one primary shard is not allocated to any node.

If you see below the health is yellow and alert is being highlighting an issue, which might usually mean that the primary shards are allocated to nodes but replicas aren’t.

A Sample kibana ES instance cluster health dash app/monitoring |||| Alert on kibana dashbaord |||| xyzIndex that was created in Part 1 is in yellow state\

Or if we do not enable the health monitoring dashboard, alternatively we can do GET call to see the health of the full cluster

GET /_cluster/health

API can be executed for multiple indices

GET /_cluster/health/employee,xyzindex

Also can be executed at shard level

GET /_cluster/health/employee?level=shards

How to resolve unassigned shards in Elasticsearch

Sometimes it could be due to late initialisation, but in our case the index is initialised and was able to insert docs to ES. But its in yellow due to some unassigned shards from the index (xyzindex) created during part 1.

Steps to identify the issue and fixing.

Identify problematic Shards

1. In my case since I was using both ES cloud dashboard and Kibana health check dashboard.

Kibana dashboard showing 0 1 & 2 as unassigned shards for xyzIndex |||| Elastic search cloud indicating issues with Shards

1. Alternatively for on premise instance, We can query by API either form local and for cloud generate API key as its needed to run the API’s

curl -XGET http://<elastichost>:9200/_cluster/health?pretty

Running from any console with API Key

curl -k -X GET -H "Authorization: ApiKey $ECE_API_KEY" https://$elastichost:9200/_cluster/health?pretty

Or simply list all the shards to identify

GET /_cat/shards

API results for _cluster/health |||| _cat/shards

Fix by either deleting those unassigned shards or by Resetting the replicas for the affected Index for any misconfigured Nodes → Shards → Replica

In our case the replicas are set to 1, as the number of nodes in the cluster is only 2 and by default when the index was created ( refer Part 1), the number of shards were set to 3 and replicas were set to 2. Basically too many shards and not enough nodes.

To resolve this issue, we can either reduce the number of replicas or add more data nodes to the cluster.

I chose the former & we can take appropriate call based on the cluster and infra architecture. In our example, we either need to add at least one more node in the cluster or reduce the replication factor to one.

Replicas are just copies of the shards which means that cluster must have the main and a replicated copy of the shard available to be in the green state

curl -k -X GET -H "Authorization: ApiKey KEY==" https://$HOST:9243/_settings?pretty" -H 'Content-Type: application/json' -d ' { "number_of_replicas": 1 }'

Elastic Search Cloud (API Console ) :: xyzindex settings were updated to reflect replcias to 1

Before (yellow) & After (green)

Conclusion:

1. ES Cloud eases the pain of setup and few button clicks away from managing the entire ES stack.
2. If we make use of the dashboards and health check it could help effectively architecture of the elastic search cluster to have meaningful configurations of nodes to Shards to replicas ratio that helps in the long run.

More on advanced queries, detecting anomalies using machine learning etc. to be continued in Part 3.

References : cloud.elastic.co