Beats — Lightweight Data Shippers

Beats are open-source data shippers that are installed on the servers to send operational data to ELK stack. They are designed to be lightweight and efficient data agents that run on servers to collect, parse and send data to centralized system like Elasticsearch. They can send data directly to Elasticsearch or via Logstash to Elasticsearch, for further processing and enhancement of data, before they are sent to Kibana for visualization.

They have a small footprint so they do not affect the performance of the system they are running on.

Elastic provides separate Beats for different types of data, depending on what data needs to be collected different beat/shipper needs to be installed on the host.

Each beat have a specific purpose, different types of Beats are mentioned below:

• Auditbeat — collect audit beat.
• Filebeat — collect log files and journals.
• Heartbeat — check the availability.
• Metricbeat — check the metrics.
• Packetbeat — for network traffic.
• Winlogbeat — collect windows event logs.

Beats in ELK

Auditbeat

Auditbeat is a lightweight shipper to audit the activities of the users and processes on the systems. Using auditbeat, we can also track changes made in the critical files like binaries or configuration files.

Filebeat

Filebeat is a lightweight shipper that forwards and centralizes the log data, it monitors the log files or the location specified, collects the log events and forward them to either Elasticsearch or Logstash.

It was designed to have a low memory footprint, handles large bulks of data, supports encryption and deal with back pressure like during ingestion process there occurs some issue with the output Filebeat will slow down the readings of the file or if there occurs some issue with network during transmission then it will remember where it left off and start from only there after the connection is re-established.

Heartbeat

Heartbeat is a lightweight daemon installed on a remote server to periodically check the status of the services and determine their availability and also tells whether the services are reachable or not. It also helps in verifying that we are meeting the SLA (service level agreement) for service uptime, security use-cases.

Metricbeat

Metricbeat is a lightweight shipper that collects the metric periodically from the operating system and the services running on the server. It takes the metrics and statistics that it collects and ships them to the output specifies such as Elasticsearch or Logstash.

It helps in monitoring the servers by collecting the metrics and services running on the server such as Apache, MongoDB, MySQL, Redis.

Metricbeat tells whether the servers are up or down but Heartbeat tells whether the services are reachable or not.

Packetbeat

It is a real-time network packet analyzer that provides an application monitoring and performance analysis. It captures the network traffic between the application server and the application layer protocols. It can also analyze issues related to application, its performance and bugs in them. It supports protocols such as ICMP, DHCP, DNS, HTTP, TLS, Redis, MySQL.

Winlogbeat

It ships windows event logs to Elasticsearch or Logstash. It can capture event data from any event logs running on the system such as application events, hardware events, security events, system events.

They are all based on the libbeat framework.

Elastic Beats make data collection simple and efficient by providing lightweight agents that can run on different systems and forward logs and metrics to Elasticsearch. They play an essential role in building a reliable logging and monitoring pipeline.

If you found this article helpful and interesting, consider exploring Logstash and Elastic Agent to understand the complete Elastic data pipeline.