ELK Stack —Curator

Many of us have faced the problem where we want to use paid features of Elasticsearch like X-Pack but don't want to pay. In this article, I am focussing on an alternative to such a paid feature of Elasticsearch’s ILM( Index Lifecycle Management), Curator. As per Elasticsearch Community, Elasticsearch has provided Index Lifecycle Management to users with at least a Basic license, beginning with Elasticsearch version 6.6 but what if you are running your ELK on some prior version. One option is to upgrade the whole stack or simply Just go for the curator!

If you are running out of disk space as you begin storing more data for longer periods of time in your ES cluster and want to automate the process of clearing out old data, Curator is a one-stop solution for you.

Curator(noun)- a keeper or custodian of a museum or other collection. Just like this, there is a curator for your ES indices too. Curator helps you in managing your Elasticsearch indices by:

1. Preparing the actionable list by collecting the full list of indices (or snapshots) from the elasticsearch cluster.
2. Iterate through a list of user-defined filters to progressively remove indices (or snapshots) from this actionable list.
3. Perform various actions on the index in an actionable list.

PREREQUISITES:

1. Running ELK Stack with the Elasticsearch ≥ 5.x versions.
2. Curator for Elasticsearch is written in Python. The Elasticsearch Curator Python API supports Python versions 2.7 and later.

Installing Curator

There are multiple ways to install Curator. Here I am going with the easiest one, Python pip.

1.) Install Python pip: $ sudo apt-get install python-pip

2.) Install Curator: $ sudo pip install elasticsearch-curator

3.) Create a directory to store all the config files of Curator $ mkdir Curator $ cd Curator

4.) let’s create the config file of Curator: $ vi curator-conf.yml

# Remember, leave a key empty if there is no value. None will be a string,
# not a Python “NoneType”
client:
 hosts:
 — 127.0.0.1
 port: 9200
 url_prefix:
 use_ssl: False
 certificate:
 client_cert:
 client_key:
 ssl_no_validate: False
 http_auth: 
 timeout: 30
 master_only: False

logging:
 loglevel: INFO
 logfile:
 logformat: default
 blacklist: [‘elasticsearch’, ‘urllib3’]

5.) Now let’s create an actions file, here you will define what data to delete and the associated retention period: $ vi curator-actions-file.yml

— -
# Remember, leave a key empty if there is no value. None will be a string,
# not a Python “NoneType”
actions:
 1:
 action: delete_indices
 description: >-
 Delete indices older than 30 days (based on index name), for deeplink-
 prefixed indices. Ignore the error if the filter does not result in an
 actionable list of indices (ignore_empty_list) and exit cleanly.
 options:
 ignore_empty_list: True
 timeout_override:
 continue_if_exception: False
 disable_action: False
 filters:
 — filtertype: pattern
 kind: prefix
 value: deeplink-*
 exclude:
 — filtertype: age
 source: name
 direction: older
 timestring: ‘%Y.%m.%d’
 unit: days
 unit_count: 30
 exclude:
 2:
 action: delete_indices
 description: >-
 Delete indices older than 60 days (based on index name), for filebeat-
 prefixed indices. Ignore the error if the filter does not result in an
 actionable list of indices (ignore_empty_list) and exit cleanly.
 options:
 ignore_empty_list: True
 timeout_override:
 continue_if_exception: False
 disable_action: False
 filters:
 — filtertype: pattern
 kind: prefix
 value: filebeat-*
 exclude:
 — filtertype: age
 source: name
 direction: older
 timestring: ‘%Y.%m.%d’
 unit: days
 unit_count: 30
 exclude:

In the above .yml file you need to manipulate the indices pattern prefix, unit_count and timestring: ‘%Y.%m.%d’ according to your ES cluster.

Not sure what pattern the indices are using? Run this command:

$ curl -XGET ‘localhost:9200/_cat/shards?pretty’

6.) This is it, you are done! Lets give it a dry run : $ sudo curator –config /home/user/Curator/curator-conf.yml –dry-run /home/user/Curator/curator-actions-file.yml

You will see something like this:

7.) So, Everything is working as expected, now run the same command removing option : — dry-run. Make sure you really want to delete the indices this time. $ sudo curator –config /home/user/Curator/curator-conf.yml /home/user/Curator/curator-actions-file.yml

This will remove your indices finally.

8.) Last step is to automate the process of running this command. You can create a shell script or just add this command in simple CRON JOB: $ crontab -e

To run the cron Job at midnight.

*0 0 sudo curator — config /home/user/Curator/curator-conf.yml /home/user/Curator/curator-actions-file.yml**

Great ! You have successfully saved your disk now from being flooded from older indices.

Happy Learning.

Reference: https://www.elastic.co/guide/en/elasticsearch/client/curator/current/index.html