Deploying ELK Stack Natively on macOS (Without Docker) — A Complete Guide for Apple Silicon

A step-by-step guide to setting up Elasticsearch, Logstash, Kibana, and Filebeat on macOS with Apple Silicon, including real-world Express.js application logging

If you’ve ever tried to set up the ELK (Elasticsearch, Logstash, Kibana) stack for log management, you’ve probably noticed that most tutorials assume you’re using Docker or Linux. But what if you want to run the stack natively on your Mac for development? What if you’re on an M1, M2, or M3 Mac and keep running into compatibility issues?

I spent a night getting this working on my M3 MacBook Air, and I’m sharing everything I learned — including the gotchas that took hours to figure out.

Why Native Installation?

While Docker is great for production, running ELK natively offers several advantages for development:

• Lower resource overhead — No Docker daemon eating RAM
• Easier debugging — Direct access to logs and configs
• Better IDE integration — Native file paths work seamlessly
• Learning opportunity — Understand how each component works

Prerequisites

Before we start, make sure you have:

• macOS (this guide tested on macOS Tahoe with M3 chip)
• Homebrew installed (brew --version to verify)
• At least 8GB RAM (16GB recommended)
• Terminal access

Architecture Overview

Here’s what we’re building:

Part 1: Installing Elasticsearch

Step 1: Tap the Elastic Repository

bash

brew tap elastic/tap

Step 2: Install Elasticsearch

bash

brew install elastic/tap/elasticsearch-full

After installation, note the important paths:

• Config: /opt/homebrew/etc/elasticsearch/
• Data: /opt/homebrew/var/lib/elasticsearch/
• Logs: /opt/homebrew/var/log/elasticsearch/

Step 3: Fix Apple Silicon Compatibility (CRITICAL!)

If you try to start Elasticsearch on an M1/M2/M3 Mac, you’ll likely see this error:

ElasticsearchException: Failure running machine learning native code.
This could be due to running on an unsupported OS or distribution...

The fix: Disable the ML module. Edit the config:

bash

nano /opt/homebrew/etc/elasticsearch/elasticsearch.yml

Add this line at the end:

yaml

xpack.ml.enabled: false

Save and exit (Ctrl + X, Y, Enter).

Step 4: Start Elasticsearch

bash

/opt/homebrew/opt/elasticsearch-full/bin/elasticsearch

Wait about 30 seconds, then verify it’s running:

bash

curl http://localhost:9200

You should see:

json

{
  "name" : "your-mac.local",
  "cluster_name" : "elasticsearch",
  "version" : { "number" : "7.17.4" },
  "tagline" : "You Know, for Search"
}

🎉 Elasticsearch is running! Keep this terminal open.

Part 2: Installing Kibana

Open a new terminal tab and run:

bash

brew install elastic/tap/kibana-full

Start Kibana:

bash

/opt/homebrew/opt/kibana-full/bin/kibana

Wait 30–60 seconds, then open http://localhost:5601 in your browser.

You should see the Kibana welcome screen!

Note: You might see some warnings about _ml index not found — this is expected since we disabled ML in Elasticsearch.

Part 3: Installing Logstash

Here’s where things get interesting. The Homebrew formula for Logstash might fail with:

Error: elastic/tap/logstash-full: undefined method 'plist_options'

The solution: Install manually.

Step 1: Download Logstash

bash

cd ~/Downloads
curl -O https://artifacts.elastic.co/downloads/logstash/logstash-7.17.4-darwin-aarch64.tar.gz
tar -xzf logstash-7.17.4-darwin-aarch64.tar.gz
sudo mv logstash-7.17.4 /opt/logstash

Step 2: Create Pipeline Config

bash

mkdir -p /opt/logstash/config
nano /opt/logstash/config/logstash-simple.conf

Add this configuration:

ruby

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "filebeat-%{+YYYY.MM.dd}"
  }
}

Step 3: Start Logstash

bash

/opt/logstash/bin/logstash -f /opt/logstash/config/logstash-simple.conf

Wait for the “Successfully started Logstash API endpoint” message.

Part 4: Installing Filebeat

Step 1: Download Filebeat

bash

cd ~/Downloads
curl -O https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-7.17.4-darwin-aarch64.tar.gz
tar -xzf filebeat-7.17.4-darwin-aarch64.tar.gz
sudo mv filebeat-7.17.4-darwin-aarch64 /opt/filebeat

Step 2: Configure Filebeat

bash

sudo nano /opt/filebeat/filebeat.yml

Find and modify these sections:

Enable the input:

yaml

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  enabled: true
  paths:
    - /var/log/*.log

Comment out Elasticsearch output:

yaml

#output.elasticsearch:
#  hosts: ["localhost:9200"]

Enable Logstash output:

yaml

output.logstash:
  hosts: ["localhost:5044"]

Step 3: Fix Permissions (CRITICAL!)

Filebeat requires the config file to be owned by root:

bash

sudo chown root:wheel /opt/filebeat/filebeat.yml

Without this, you’ll see:

Exiting: error loading config file: config file must be owned by root

Step 4: Start Filebeat

bash

sudo /opt/filebeat/filebeat -e -c /opt/filebeat/filebeat.yml

Look for “Connection to backoff(async(tcp://localhost:5044)) established” — that means it’s connected!

Part 5: Viewing Logs in Kibana

Create an Index Pattern

1. Go to http://localhost:5601
2. Navigate to Stack Management → Index Patterns
3. Click Create index pattern
4. Enter filebeat-* as the pattern
5. Select @timestamp as the time field
6. Click Create

View Your Logs

1. Click the hamburger menu (☰)
2. Go to Discover
3. You should see logs flowing in!

Part 6: Connecting a Real Express.js Application

Now, let’s connect a real application. I’ll show you how to set up ECS-formatted logging with Pino.

Step 1: Set Up Your Logger

Install dependencies:

bash

npm install pino @elastic/ecs-pino-format

Create logger.ts:

typescript

import pino from 'pino';
import ecsFormat from '@elastic/ecs-pino-format';
const logger = pino({
  ...ecsFormat({
    serviceName: 'my-express-api',
    serviceVersion: '1.0.0',
    serviceEnvironment: process.env.NODE_ENV || 'development',
  }),
  level: process.env.LOG_LEVEL || 'info',
});
export default logger;

Step 2: Pipe Logs to a File

Run your application with output piping:

bash

mkdir -p ~/app-logs
NODE_ENV=production npm run dev 2>&1 | tee -a ~/app-logs/app.log

Step 3: Update Filebeat Config

Add your app logs to Filebeat with JSON parsing:

bash

sudo nano /opt/filebeat/filebeat.yml

Add a new input:

yaml

- type: filestream
  id: app-logs
  enabled: true
  paths:
    - /Users/yourusername/app-logs/*.log
  parsers:
    - ndjson:
        target: ""
        add_error_key: true

Restart Filebeat after the change.

Step 4: View Application Logs in Kibana

Now in Kibana Discover, you can:

• Filter by service.name: "my-express-api"
• Search by log.level: "error" to find errors
• Filter by event.action for specific events
• See full request/response details with ECS fields

Troubleshooting Common Issues

Issue: “Failure running machine learning native code”

Solution: Add xpack.ml.enabled: false to elasticsearch.yml

Issue: Homebrew Logstash formula fails

Solution: Download and install manually from Elastic’s website

Issue: “Config file must be owned by root”

Solution: sudo chown root:wheel /opt/filebeat/filebeat.yml

Issue: Filebeat can’t connect to Logstash

Solution: Make sure output.logstash uses just localhost:5044 (no http:// prefix)

Issue: Logs appear but fields aren’t parsed

Solution: Add ndjson parser to Filebeat input, ensure your app outputs JSON (not pretty-printed)

Starting/Stopping the Stack

Here’s a handy script to manage everything:

bash

#!/bin/bash
# save as ~/elk-stack.sh
case "$1" in
  start)
    echo "Starting Elasticsearch..."
    /opt/homebrew/opt/elasticsearch-full/bin/elasticsearch &
    sleep 30
    echo "Starting Kibana..."
    /opt/homebrew/opt/kibana-full/bin/kibana &
    sleep 10
    echo "Starting Logstash..."
    /opt/logstash/bin/logstash -f /opt/logstash/config/logstash-simple.conf &
    sleep 30
    echo "Starting Filebeat..."
    sudo /opt/filebeat/filebeat -e -c /opt/filebeat/filebeat.yml &
    echo "ELK Stack started!"
    ;;
  stop)
    echo "Stopping ELK Stack..."
    pkill -f elasticsearch
    pkill -f kibana
    pkill -f logstash
    sudo pkill -f filebeat
    echo "ELK Stack stopped!"
    ;;
  *)
    echo "Usage: $0 {start|stop}"
    exit 1
esac

Make it executable:

bash

chmod +x ~/elk-stack.sh

Summary

We’ve successfully set up a complete ELK stack on macOS with Apple Silicon:

ComponentPortPurposeElasticsearch9200Data storage & searchKibana5601Visualization UILogstash5044Log processingFilebeat — Log shipping

Key takeaways:

1. Disable ML for Apple Silicon — The xpack.ml.enabled: false setting is crucial
2. Manual installation may be needed — Homebrew formulas can have compatibility issues
3. Permissions matter — Filebeat requires root-owned config files
4. JSON logging is essential — Use ECS format for proper field parsing
5. ndjson parser — Enable it in Filebeat for JSON log parsing

What’s Next?

Now that you have ELK running locally, you can:

• Build dashboards — Create visualizations for your app metrics
• Set up alerting — Get notified when errors spike
• Add more data sources — Monitor multiple applications
• Explore APM — Add Elastic APM for performance monitoring

Found this helpful? Follow me for more DevOps and full-stack development content!

Tags: #elasticsearch #kibana #logstash #macos #devops #logging #nodejs #expressjs #applesilicon

This guide was tested on macOS Tahoe with Apple M3 chip, Elasticsearch 7.17.4, Kibana 7.17.4, Logstash 7.17.4/9.2.3, and Filebeat 9.2.3.