Implementing a Secure Email Server in a Test ENV

Tired of phishing and spoofing? Learn how to deploy your own secure email server and implement the magic trinity of email authentication.

Introduction

Every day, billions of emails crisscross the globe, carrying everything from holiday photos to sensitive business documents. Yet, the underlying protocol, SMTP (Simple Mail Transfer Protocol), was designed in an era of trust, with little built-in security. This has turned our inboxes into a Wildlife-like, rife with threats:

• Phishing: Deceptive emails impersonating your bank, Netflix, or a colleague, designed to steal your credentials.
• Spoofing: Forging the “From” address to make a malicious email appear from a legitimate source.
• Spam: The endless deluge of unsolicited messages, clogging productivity.
• Malware: Emails with malicious attachments or links that can infect your systems.

The Trio of Trust: SPF, DKIM, and DMARC

Before we start building, let’s understand the essential technologies that act as bouncers for your email domain.

• SPF (Sender Policy Framework): This is a DNS record that acts like a guest list. It publicly declares which mail servers are authorized to send emails from your domain. If an email comes from a server not on the list, receiving servers can mark it as suspicious.
• DKIM (DomainKeys Identified Mail): This is your digital signature. It adds a cryptographic signature to the headers of every outgoing email. The receiving server uses a public key published in your DNS to verify that the email was sent by you and hasn’t been tampered with in transit.
• DMARC (Domain-based Message Authentication, Reporting & Conformance): This is the policy enforcer. DMARC tells receiving servers what to do if an email fails SPF and/or DKIM checks (e.g., quarantine it or reject it outright). Crucially, it also provides you with forensic reports, showing you who is trying to send email using your domain.

Together, these three protocols form a powerful defense, making it incredibly difficult for attackers to spoof your domain and ensuring your legitimate emails are delivered.

Building a Multi-Layered Defense

A truly secure email system requires a defense-in-depth approach, combining proactive authentication with reactive threat detection. While protocols like SPF, DKIM, and DMARC verify an email’s origin and integrity, they don’t scan its content for malicious intent or payloads. This is where content-filtering tools become essential:

• SpamAssassin: This is your intelligent spam filter. Using a complex rule-based system, Bayesian filtering, and network checks, it analyzes email content, headers, and sending patterns to identify and block spam with remarkable accuracy. It learns from your email habits, constantly adapting to new spamming techniques.
• ClamAV: As the guardian against digital pathogens, this open-source antivirus engine meticulously scans all email attachments for viruses, trojans, malware, and other malicious software. It ensures that no harmful payload enters your system through the email gateway.

iRedMail

Manually configuring an email server (Postfix for SMTP, Dovecot for IMAP, SpamAssassin, ClamAV, etc.) is a complex and time-consuming task. This is where iRedMail shines.

iRedMail is an open-source project that automates the installation and configuration of a full-featured, secure mail server on a Linux system. it gives you a production-ready system with:

• A mail transfer agent (Postfix)
• An IMAP/POP3 server (Dovecot)
• A webmail client
• Spam and virus filtering (SpamAssassin, ClamAV)
• An administrative panel

Implementation

A Critical Note on Domain Names: For a production server, you must have a real domain name (e.g., yourcompany.com) to create the necessary DNS records (MX, A, SPF, DKIM, DMARC). However, for learning and testing, we can simulate this environment locally.

Step 1: Provision The Server

We’ll use a fresh Ubuntu virtual machine. Using a VM (on VMware) for isolation.

1. Update the system:

sudo apt update && sudo apt upgrade -y

Updating the system

2. Set a hostname. Since we’re testing, we’ll use mail.test.local:

sudo hostnamectl set-hostname mail.test.local

Setting up the hostname

3. Edit the hosts file to simulate DNS for our fake domain. Open /etc/hosts and add:

127.0.0.1   localhost
127.0.0.1   mail.test.local

Configuring /etc/hosts file

Step 2: iRedMail Installation

1. Download the latest stable release of iRedMail from their official website.

Downloading the latest stable release of iRedMail

2. Extract the tarball and run the installer:

tar xvf *.tar.gz
cd iRedMail-*/
chmod +x iRedMail.sh
sudo bash iRedMail.sh

Extracting the tarball

Running the installer

3. The installer will present a straightforward text-based wizard. Here are the key choices:

• Mail Storage Path: Choose /var/vmail.

Setting the Mail Storage Path

• Web Server: Choose Nginx.

Setting the Web Server

• Backend: Choose MySQL/MariaDB.

Setting the Backend

• First Mail Domain Name: Enter test.local.

Setting the First Mail Domain Name

• Password for the postmaster@test.local account: Choose a very strong password and save it!
• Optional Components: Use The default.

The installer will run for several minutes. Once finished, reboot the server.

Step 3: Accessing The New Email Server

After the reboot, you can access your new email server:

• Webmail: Open https://mail.test.local/mail in your browser. Log in as postmaster@test.local.

Webmail 1

Webmail 2

• Admin Panel: Open https://mail.test.local/iredadmin. This is where you manage domains, users, and policies.

Admin Panel 1

Admin Panel 2

Trying to send an email between two user accounts we have created.

Sending an email between 2 users — sending

Sending an email between 2 users — Receiving

Step 4: Implementing the Security Trio

Note:

So far, our test server can send and receive mail internally, but our SPF, DKIM, and DMARC setup is untested. Why? Because these technologies are built on DNS — a public directory where our fake test.local domain doesn't exist.

To close this gap, we will install a local DNS server. This allows us to host the required records ourselves, creating a self-contained network where email authentication can be fully validated. This is the final piece that makes our test environment behave like the real internet.

Setting Up Local DNS :

1. Install and Configure dnsmasq

sudo apt install dnsmasq -y
sudo systemctl stop dnsmasq

Installing dnsmasq

2. Get The Actual DKIM Key

sudo amavisd showkeys

Getting The Actual DKIM Key

3. Configure dnsmasq

sudo mv /etc/dnsmasq.conf /etc/dnsmasq.conf.backup
sudo nano /etc/dnsmasq.confcontent

Configuring dnsmasq

# Content

# Basic dnsmasq configuration for email lab
interface=lo
bind-interfaces

# Local domain definition
domain=test.local
local=/test.local/

# A records - point domains to your server
address=/test.local/127.0.0.1
address=/mail.test.local/127.0.0.1

# MX record - define mail server priority
mx-host=test.local,mail.test.local,10

# SPF record
txt-record=test.local,"v=spf1 mx ~all"

# DKIM record (get this from your iRedMail setup)
txt-record=default._domainkey.test.local,"v=DKIM1; k=rsa; p=YOUR_ACTUAL_DKIM_PUBLIC_KEY_HERE"

# DMARC record
txt-record=_dmarc.test.local,"v=DMARC1; p=none; rua=mailto:postmaster@test.local"

# PTR record for reverse DNS (important for some spam checks)
ptr-record=1.0.0.127.in-addr.arpa,"mail.test.local"

# Local cache settings
cache-size=1000
local-ttl=300

4. Configure System to Use Local DNS

sudo nano /etc/resolv.conf

# Content

nameserver 127.0.0.1
domain test.local
search test.local

Important: Make this permanent to prevent network manager from overwriting it:

sudo chattr +i /etc/resolv.conf

5. Start DNS Service

sudo systemctl start dnsmasq
sudo systemctl enable dnsmasq

Starting DNS Service

Step 5: Testing

1. Verify DNS Records are Working

# Test A record
nslookup mail.test.local

# Test MX record
nslookup -type=mx test.local

# Test SPF record
nslookup -type=txt test.local

# Test DKIM record  
nslookup -type=txt default._domainkey.test.local

# Test DMARC record
nslookup -type=txt _dmarc.test.local

Testing A and MX record

Testing SPF and DKIM record

Testing DMARC record

Testing SPF — 127.0.0.1 point to localhost (not authorized)

Testing SPF — mail.test.local (authorized)

Testing End-to-End Authentication (DKIM Signing)

Note:

The absence of SPF and DMARC results in these headers is expected. These checks are primarily enforced when receiving mail from external servers to prevent forgery. For internal mail between users on the same server, iRedMail correctly prioritizes deliverability and speed, relying on the fact that the users are already authenticated. Our setup is still correct; the authentication will be fully evaluated when sending to or receiving from the outside world.

Step 5: Implementing SpamAssassin & ClamAV

Note:

Before we proceed, it’s important to know that SpamAssassin (for spam) and ClamAV (for viruses) are not additional packages we need to install. They are core components of the iRedMail ecosystem and were installed and activated during the initial setup. Our focus now shifts to validation and tuning, ensuring these powerful tools are configured to our specific needs.

1. Optimizing SpamAssassin Configuration

Enable Bayesian Filtering (Machine Learning)

sudo nano /etc/spamassassin/local.cf

Add or modify these lines:

# Bayesian filtering (learns from your email patterns)
use_bayes 1
bayes_auto_learn 1
bayes_auto_learn_threshold_nonspam 0.1
bayes_auto_learn_threshold_spam 6.0

# Network checks (requires internet)
skip_rbl_checks 0
use_razor2 1
use_dcc 1
use_pyzor 1

# Score adjustments
required_score 5.0
rewrite_header Subject *****SPAM*****

Update SpamAssassin Rules

sudo sa-update

Updating SpamAssassin Rules

2. Optimizing ClamAV Configuration

Configure Freshclam (Virus Definition Updates)

sudo nano /etc/clamav/freshclam.conf

Ensure these settings:

# Update checks (24 times per day)
Checks 24
# Use multiple databases
DatabaseMirror database.clamav.net

Optimize ClamAV Daemon

sudo nano /etc/clamav/clamd.conf

Important settings:

# Scan within archives
ScanArchive yes
ScanOLE2 yes
ScanPDF yes
ScanHTML yes
# Maximum file size to scan (100MB)
MaxFileSize 100M
MaxScanSize 100M

3. Testing SpamAssassin & ClamAV

Test 1: Send EICAR Test Virus

Test file

The email is sent from user1 to user2

The user2 didn’t receive the email

The email is filtered and findings are reported to admin

Test 2: Send GTUBE Test Spam

Test email

Email is detected as a spam

Conclusion

By implementing iRedMail with SPF, DKIM, and DMARC, we’ve built a secure, self-hosted email server that authenticates senders and prevents domain spoofing. Our test environment, complete with local DNS, proved the system works, while the integrated SpamAssassin and ClamAV provide robust protection against spam and malware. This journey demonstrates that achieving email sovereignty is entirely possible.

You now possess the knowledge and a working foundation to deploy a private, secure email platform, moving from a passive user to an empowered administrator in control of your own digital communication.