Photo by Maxim Melnikov on Unsplash

Logstash Multiple Pipeline Configuration

Most of us are already familiar with the Elastic products. In this blog, we’ll walk through the steps for configuring Logstash to ingest and process data using multiple pipelines.

Logstash is an open-source data processing pipeline that allows you to ingest, transform, and output data. It’s a powerful tool for collecting and processing large amounts of data from various sources, making it useful for managing and analyzing logs, metrics, and other types of data.

Installation steps

Since installing the Logstash is pretty straightforward, we will directly jump into the configuration part.

For installation, please follow — https://www.elastic.co/guide/en/logstash/current/installing-logstash.html

Configuration

For today, we will test the config for Syslog and File inputs. To achieve this, first, you’ll need to create two configuration files, one for each pipeline.

Step 1: Create Input Configuration Files

For deb/rpm package, go to /etc/logstash/conf.d/ directory and create two files : syslog.conf and file.conf.

cd /etc/logstash/conf.d
touch syslog.conf file.conf

Step 2: Configure the Syslog Pipeline

In the syslog.conf file, add the following configuration:

input {
  syslog {
    port => 514
  }
}
filter {
  # Add any filters you need here
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "firewall-%{+YYYY.MM.dd}"
  }
}

This configuration specifies that Logstash should listen for syslog data on port 514, apply any filters you need, and then send the output to Elasticsearch with an index name based on the current date.

Step 3: Configure the File Input Pipeline

In the file.conf file, add the following configuration:

input {
  file {
    path => "/var/log/syslog"
  }
}
filter {
  # Add any filters you need here
}
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "syslog-%{+YYYY.MM.dd}"
  }
}

This configuration specifies that Logstash should read log files from the /var/log/syslog directory, apply any filters you need, and then send the output to Elasticsearch with an index name based on the current date.

Step 4: Update Configuration in Pipeline YML

Add the path of the above configurations in pipeline.yml located in /etc/logstash/ directoty.

- pipeline.id: syslog
  path.config: "/etc/logstash/conf.d/syslog.conf"

- pipeline.id: file-input
  path.config: "/etc/logstash/conf.d/file.conf"

Step 5: Start Logstash with Multiple Pipelines

To start Logstash with multiple pipelines, start the service and validate the output for any error (if any)

systemctl start logstash
sudo tail -f /var/log/logstash/logstash-plain.log

Logstash will automatically detect and load all configuration files in the specified directory. It will also create separate pipelines for each configuration file.

Tip: Before running the logstash service, we can even test our Logstash configuration using below command

sudo /usr/share/logstash/bin/logstash --config.test_and_exit -f <path_to_config_files>

In conclusion, Logstash’s support for multiple pipelines allows you to process different types of data using different configurations.

Cheers!