ElasticFrom Custom Log Scripts to Scalable Logging: Implementing Elasticsearch with Filebeat & Keystore

This article explains how we migrated from a custom log-pushing script to a scalable logging solution using Filebeat and Elasticsearch. It highlights the challenges we faced with memory usage, reliability, and security, and how Filebeat with Keystore solved them efficiently. A practical step-by-step implementation guide is included.

Logging is easy — until it isn’t.

At first, we built a simple command-line script that:

1. Read log files
2. Parsed data
3. Pushed logs to Elasticsearch

It worked… until production traffic increased.

That’s when the real problems started.

🚨 The Problem with Custom Log Scripts

Our old approach looked something like this:

php read_logs.php

Inside the script:

• Read file using file_get_contents()
• Parse JSON
• Send via CURL to Elasticsearch

It had multiple issues:

❌ 1. High Memory Usage

Large files were loaded entirely into memory.

❌ 2. No Real-Time Streaming

We had to run cron jobs repeatedly.

❌ 3. Failure Handling Was Weak

If Elasticsearch was down → logs were lost.

❌ 4. Security Risk

Credentials were stored directly in the script:

$auth = "elastic:password123";

That’s dangerous in production.

We needed something:

• Lightweight
• Real-time
• Secure
• Production-ready

That’s when we moved to Filebeat.

🚀 What is Filebeat?

Filebeat is a lightweight log shipper from Elastic Stack.

It:

• Watches log files
• Detects new lines
• Sends them to Elasticsearch
• Handles retries automatically
• Keeps track of file offsets

Basically, it does everything our custom script was trying to do — but better.

🛠 Step-by-Step: Implementing Filebeat

1️⃣ Install Filebeat

On Ubuntu:

sudo apt update
sudo apt install filebeat

Check version:

filebeat version

2️⃣ Configure Filebeat

Edit:

/etc/filebeat/filebeat.yml

Example Configuration:

filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/application/*.log

output.elasticsearch:
  hosts: ["https://localhost:9200"]
  username: "elastic"
  password: "your_password"

Now Filebeat watches logs in real-time.

🔐 Problem: Hardcoding Credentials (Bad Practice)

Storing passwords inside filebeat.yml is risky.

Instead, use Filebeat Keystore.

🔐 What is Filebeat Keystore?

It securely stores secrets (like passwords, API keys).

No plain-text credentials inside config files.

🛠 How to Implement Filebeat Keystore

1️⃣ Create Keystore

sudo filebeat keystore create

2️⃣ Add Elasticsearch Password

sudo filebeat keystore add ES_PASSWORD

It will prompt for password securely.

3️⃣ Update filebeat.yml

Instead of:

password: "your_password"

Use:

password: "${ES_PASSWORD}"

Now password is securely loaded from keystore.

4️⃣ Test Configuration

sudo filebeat test config

5️⃣ Start Filebeat

sudo systemctl start filebeat

Enable on boot:

sudo systemctl enable filebeat

🧠 What Problems Did This Solve?

Before:

• Manual Script
• Memory-Heavy
• Cron Dependency
• Credentials Exposed
• No Retry Handling

After:

• Automatic Log Steaming
• Lightweight Agent
• Real-Time Monitoring
• Secure Keystore
• Built-in Retry & Buffering

🎯 Architecture Overview

Application → Log File → Filebeat → Elasticsearch → Kibana

Simple. Scalable. Secure.

⚡ Bonus: Why Filebeat is Better Than Custom Scripts

• Uses minimal CPU
• Handles file rotation
• Tracks file offset
• Supports multiline logs
• Built-in backpressure handling
• Production tested

Your custom script would eventually re-implement half of Filebeat badly.

🏁 Final Thoughts

Moving from a custom log-push command to Filebeat was a major upgrade.

It improved:

✔ Reliability ✔ Security ✔ Scalability ✔ Maintainability

Sometimes the best engineering decision is not writing more code — it’s using the right tool.