Centralized Logs & Routing them: rsyslog engine on Docker

rsyslog is a fast & efficient tool for ingesting and routing event logs.

So, let’s setup a basic central log server in a few steps, using rsyslog’s official docker image.

First, create a directory in the home folder for the log file and an rsyslog configuration file that we will later bind mount to the container:

mkdir ~/rsyslog
cd ~/rsyslog
mkdir rsyslog-logs
cat > rsyslog.conf

And the rsyslog.conf file that will write all logs to our desired log file:

module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

*.* /var/log/remote/syslog.log

Keep in mind that normally rsyslog on Linux systems use Unix sockets to write logs to /var/log/rsyslog file, but this configuration works fine for our container.

Next, run the docker container, I’ll simply bind the config file of rsyslog and the directory that will keep our log file:

docker run -d \
  --name rsyslog \
  -p 514:514/tcp \
  -p 514:514/udp \
  -v ~/rsyslog/rsyslog.conf:/etc/rsyslog.conf:ro \
  -v ~/rsyslog/rsyslog-logs:/var/log/remote \
  rsyslog/rsyslog

That’s it, now we have an rsyslog engine running on docker and listening on port 514 for logs, then it forwards them all into our syslog.log file.

Now lets send a log to port 514 to test it:

logger -n 127.0.0.1 -P 514 -t my-daemon "this is an example log"

Your log should show up under the syslog.log ~/rsyslog/rsyslog-logs/syslog.log file on the host system.

Now all you have to do is take any service, forward its logs to this rsyslog engine on port 514, just like we did with the logger command, and voila, you can do whatever you want with it.

Recall that rsyslog uses RFC 5424 format by default.

Now lets do something fun such as routing logs with syslogseverity level less than 3 to a different file. Stuff like this is brainless to do using rsyslog. Let’s also add the syslogseverity field to our logs so we can see them. For this, edit the rsyslog.conf file we previously created:

module(load="imudp")
input(type="imudp" port="514")

module(load="imtcp")
input(type="imtcp" port="514")

# Template including severity display
template(name="WithPRI" type="string"
         string="%timegenerated% %hostname% %syslogtag% [%syslogfacility%.%syslogseverity%] %msg%\n")

# --- High priority logs (severity 0,1,2) ---
*.emerg;*.alert;*.crit  action(type="omfile"
                               file="/var/log/remote/highprio.log"
                               template="WithPRI")

# --- Everything else ---
*.* action(type="omfile"
           file="/var/log/remote/syslog.log"
           template="WithPRI")

Now restart the container so the new config kicks in ;)

docker restart rsyslog

And send logs of different severities to the port 514:

logger -n 127.0.0.1 -P 514 -p user.info -t somedaemon "this is an example log"
logger -n 127.0.0.1 -P 514 -p user.crit -t somedaemon "this is an example log"

Recall that crit is a level 2 severity while info is a 6. reference:

[embed]syslogseverity - rsyslog documentation Provides the numeric syslog severity extracted from the message. This property belongs to the Message Properties group…www.rsyslog.com

This is one of many things rsyslog can do, it is absolute power tool that can handle many, many logs. Wait till I tell you about omprog or omfwd on the next article.

“journald is a bitch, all my niggas use syslog” -me