Everything about Elastic search (Basics) Part 1

The intent is to share basic understanding on elastic search and it’s ecosystem. It also covers the basics of Elastic search Query DSL for creation of Index and querying through few full text and term queries.

The flow diagram below represents the high level flow in to ELK,

• Different data sources ( Beats or Kafka), In some cases Beats data can directly be pushed to ES, if there is no processing needed to the data )
• The collected data gets processed by a data processing layer (Logstash) , FluentD can also be leveraged (alternative)
• Processed data would pushed as documents in Elastic search in the corresponding index.
• These documents will further be available in Kibana for index based search or dashboards.

Elastic stack data flow process ( Included Apache Kafka non elastic as a data source too )

Typically the elastic search will be used for app search (akin to Solr) for example product search in a e-commerce web portal. Alternatively the entire stack could be leveraged for any data processing and analysis, and also at the same time store indexed data(elastic search) for further processing into elegant out of the box dashboards on Kibana. Elastic 7.9 has more to offer interms of APM, ML, analysers among others.

kibana dashboard generated on top of sample flight data.

What, Why & How of Elastic Search?

1. Elastic search the key module in Elastic ecosystem(ELK stack et al) and is known for its simple query via REST, scalability, distributed architecture and of-course speed.
2. It could be used for any kind of structured or unstructured data to search and make meaningful analysis and it typically support JSON format.
3. the whole ELK stack becomes a one stop solution for data ingestion, reporting and visualisations.
4. Raw data ingestion is critical and thats how the data should be enriched before we index(collection of docs that we intend to store and query upon) in Elastic search, post this it is pretty straight forward to run the queries against the data and run reports or visualisations using Kibana (visualisation tool to manage).
5. Elastic Index uses inverted index which actually allows for faster full text search
6. Its available as open-source as well as the elastic cloud offering too.

Use-cases

Typical use-cases include below:

1. Fast Search : ( App data / web data/ system metrics/ logs )) for example: product search in B2C/B2B .
2. Observability : Logs search and analytics ( search and debug application logs, system metrics , APM ).
3. Visualisations : (easy reporting via visual appeal )
4. Misc. All purpose analytics : ( consider it as any other faster DB to run analytical queries to generate analytical reports ).

Getting Started:

It’s readily available on the elastic website ; please find some inferences from the site below.

Nodes vs Shards vs replica ( Dos and Don’ts)

when a new node is setup in the elastic cluster, there are few basic default config that would happen if we miss to specify

• < 7.x by default each new Index or document created would have 5 shards, 7.x onwards its defaulted to 1 ( in AWS ELK 7.x still creates 5 shards by default)
• when we index multiple docs it would get stored in any of the 5 shards
• When we add a new node in the same cluster, then the above 5 shards will be evenly shared across the two nodes
• Key takeaway: if we have single node in cluster and do not anticipate the growth of data, better stick to one Shard at the time of creation manually passing the shard details

In case of replica, its kind of shard but stores the replication of the data

• Replicas ensures increase in search performance and primarily fail-over.
• if we don’t specify at the time of index creation, the default replica is 1
• Key takeaway: the replica (shard) would always be created on another instance/node in the elastic cluster, so if the node is single node, no point in setting up the replica.

How to create Index and Documents

Create Index ( just index)

PUT xyzindex
{
    "settings" : {
        "number_of_shards" : 3,
        "number_of_replicas" : 2
    }
}

Response :
{
  "acknowledged" : true,
  "shards_acknowledged" : true,
  "index" : "xyzindex"
}

When we execute below query/request; it would automatically creates the employee index if it doesn’t exist already, and also adds a new document that has an ID of 123, and stores and indexes the name,num,addressfields. The response shows that a doc is created and a new index with 2 total shards

PUT /employee/_doc/123
{
  "name": "Pradeep",
  "num":"9000000000",
  "address" : "India, Bengaluru"
}

Response:
{
  "_index" : "employee",
  "_type" : "_doc",
  "_id" : "123",
  "_version" : 1,
  "result" : "created",
  "_shards" : {
    "total" : 2,
    "successful" : 1,
    "failed" : 0
  },
  "_seq_no" : 0,
  "_primary_term" : 1
}

Elastic Query DSL and elastic search SQL

• If we executed the PUT employee from above, new document would be available for querying. we can retrieve it with a GET request with the doc Id and the _version indicates that the doc is the 3rd version of it.

GET /employee/_doc/123

{
  "_index" : "employee",
  "_type" : "_doc",
  "_id" : "123",
  "_version" : 5,
  "_seq_no" : 6,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "name" : "Pradeep",
    "num" : "9000000000",
    "address" : "India, Bengaluru"
  }
}

• The other key advantage of elastic search is that we could leverage Elasticsearch SQL for familiar syntax

POST /_sql?format=txt
{
  "query": "SELECT * FROM employee WHERE name = 'Pradeep'"
}

    address     |     name      |      num      
----------------+---------------+---------------
India, Bengaluru|Pradeep        |9000000000

• Simple search query samples : the following would get all documents part of employee index

GET /employee/_search
{
  "query": { "match_all": { } }
}

Full Text Queries

Match Query

Standard query for full text queries

name contains pradeep

GET /employee/_search
{
  "query": {
    "match": {
      "name": {
        "query": "pradeep"
      }
    }
  }
}

address contains india

GET /employee/_search
{
  "query": {
    "match": {
      "address": {
        "query": "india"
      }
    }
  }
}

Fuzziness for Idnia to India

GET /employee/_search
{
  "query": {
    "match": {
      "address": {
        "query": "Idnia",
        "fuzziness": "AUTO"
      }
    }
  }
}

Match Phrase Query

The match_phrase query analyses the text and creates a phrase query out of the analysed text & match_phrase query does not support fuzziness

GET /employee/_search
{
  "query": {
    "match_phrase": {
      "address": {
        "query": "india"
      }
    }
  }
}

Term Level Queries

A few of the term level queries with samples are explained below. This do not analyse search terms but finds documents based on precise values in structured data for example date ranges, IP addresses, prices, names and for more details or additional queries not covered refer official docs here

Exist Query

Ability to identify if a term exist, and would give the results of docs contains the field name

Find documents which has name field

GET /employee/_search
{
  "query": {
    "exists": {
      "field": "name"
    }
  }
}

Find documents with missing field phone 

GET /employee/_search
{
  "query": {
    "bool": {
      "must_not": {
        "exists": {
          "field": "employee.phone"
        }
      }
    }
  }
}

IDs Query

Enables to query by document Id ( Single or multiple : array of document Ids)

GET /employee/_search
{
  "query": {
    "ids" : {
      "values" : ["1", "2", "123"]
    }
  }
}

Fuzzy Query

Returns documents that contain terms similar to the search term

GET /employee/_search
{
  "query": {
    "fuzzy": {
      "address": {
        "value": "India"
      }
    }
  }
}

if we want to add some additional advanced fuzziness params as below will provide more additional filters ( Please explore them )

GET /employee/_search
{
  "query": {
    "fuzzy": {
      "address": {
        "value": "India",
        "fuzziness": "AUTO",
        "max_expansions": 50,
        "prefix_length": 0,
        "transpositions": true,
        "rewrite": "constant_score"
      }
    }
  }
}

Prefix Query

Matches and returns the docs with the field starting with prefix value

GET /employee/_search
{
  "query": {
    "prefix": {
      "name": {
        "value": "p"
      }
    }
  }
}

Misc.

Mappings

GET /employee/_mapping

ES Cloud and Health is available in part 2 of the series, Advanced queries will follow in next part. To be continued.

References : https://www.elastic.co/