Setting Up Your Own Mail Server with Postfix and Dovecot

Introduction

Running your own mail server is one of those tasks that seems daunting at first, but once you break it down, it's surprisingly manageable. I've set up quite a few over the years, and in this guide, I'll walk you through the exact steps I use to get a reliable mail server up and running with Postfix and Dovecot on Debian.

"Why run your own mail server? Privacy, control, and the satisfaction of knowing exactly how your email flows. Plus, you learn an incredible amount about DNS, security, and system administration."

Why Postfix + Dovecot?

The combination of Postfix and Dovecot is the golden standard for self-hosted mail. Here's why:

• Postfix: Handles SMTP (sending and receiving mail). It's fast, secure, and well-documented.
• Dovecot: Handles IMAP/POP3 (storing and retrieving mail). It plays beautifully with Postfix.
• Both are mature, battle-tested, and work together seamlessly.

Prerequisites

Before we begin, run through this checklist to make sure you have everything ready:

• A Debian 12 server (a VPS from Hetzner or DigitalOcean works great)
• A domain name with DNS access (I'll use opama.dev as an example)
• Root or sudo access on your server
• Basic familiarity with the command line
• Ports 25, 587, 993, and 143 open on your firewall

Let's dive in.

Step 1: DNS Configuration

Before installing anything, we need to set up DNS records. This is the most critical part -- if your DNS isn't right, your mail won't flow.

A Record for the Mail Server

Make sure your server has an A record pointing to its IP:

mail.opama.dev.  IN  A  203.0.113.42

MX Record

The MX record tells the world where to deliver mail for your domain:

opama.dev.  IN  MX  10  mail.opama.dev.

The 10 is the priority. Lower values are preferred.

SPF Record

SPF (Sender Policy Framework) tells receiving servers which IPs are allowed to send mail for your domain. Without it, your emails will likely land in spam.

opama.dev.  IN  TXT  "v=spf1 mx a:mail.opama.dev -all"

DKIM Key Pair

We'll generate the DKIM key after installing OpenDKIM, but you'll publish the public key as a TXT record:

default._domainkey.opama.dev.  IN  TXT  "v=DKIM1; h=sha256; k=rsa; p=MIGfMA0GCSqGSIb4DQEBAQUAA4GNADCBiQKBgQC..."

DMARC Record

DMARC ties SPF and DKIM together and tells receivers what to do when checks fail:

_dmarc.opama.dev.  IN  TXT  "v=DMARC1; p=quarantine; rua=mailto:postmaster@opama.dev"

Step 2: Installing Postfix

Now let's get our hands dirty. SSH into your server and install Postfix:

sudo apt update && sudo apt upgrade -y
sudo apt install postfix -y

During installation, select Internet Site and set your system mail name to opama.dev.

Main Configuration

The main Postfix config lives at /etc/postfix/main.cf. Here's the setup I use:

sudo postconf -e "myhostname = mail.opama.dev"
sudo postconf -e "mydomain = opama.dev"
sudo postconf -e "myorigin = \$mydomain"
sudo postconf -e "inet_interfaces = all"
sudo postconf -e "inet_protocols = ipv4"
sudo postconf -e "mydestination = \$myhostname, localhost.\$mydomain, localhost, \$mydomain"
sudo postconf -e "home_mailbox = Maildir/"
sudo postconf -e "smtpd_banner = \$myhostname ESMTP \$mail_name"
sudo postconf -e "smtpd_tls_cert_file = /etc/letsencrypt/live/mail.opama.dev/fullchain.pem"
sudo postconf -e "smtpd_tls_key_file = /etc/letsencrypt/live/mail.opama.dev/privkey.pem"
sudo postconf -e "smtpd_tls_security_level = may"
sudo postconf -e "smtp_tls_security_level = may"

Enabling STARTTLS on Submission

We also need to enable submission on port 587 with STARTTLS:

sudo postconf -M submission/inet="submission inet n - n - - smtpd"
sudo postconf -P "submission/inet/syslog_name=postfix/submission"
sudo postconf -P "submission/inet/smtpd_tls_security_level=encrypt"
sudo postconf -P "submission/inet/smtpd_sasl_auth_enable=yes"

Step 3: Installing Dovecot

With Postfix handling message transfer, we need Dovecot to manage mailboxes and handle IMAP connections.

sudo apt install dovecot-core dovecot-imapd -y

Dovecot Configuration

Dovecot's config is spread across /etc/dovecot/. Here's what we need:

sudo tee /etc/dovecot/dovecot.conf << 'EOF'
listen = *
protocols = imap

ssl = required
ssl_cert = </etc/letsencrypt/live/mail.opama.dev/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.opama.dev/privkey.pem

mail_location = maildir:~/Maildir

service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}
EOF

The ssl = required directive means Dovecot will only accept connections over TLS. This is non-negotiable if you want to keep your users' email secure.

Step 4: Testing the Setup

Now let's verify everything works. Restart both services:

sudo systemctl restart postfix dovecot
sudo systemctl enable postfix dovecot

Sending a Test Email

From the server itself, send a test email:

echo "Testing my new mail server!" | sudo sendmail -v test@example.com

Video Walkthrough

If you prefer watching a full walkthrough, here's a great video that covers the same setup:

Checking Logs

Keep an eye on the mail logs for any issues:

sudo journalctl -u postfix -f
sudo tail -f /var/log/mail.log

You should see entries like this when mail flows through:

May 15 10:23:45 mail postfix/smtpd[1234]: connect from localhost[127.0.0.1]
May 15 10:23:45 mail postfix/smtpd[1234]: 4XYZ123: client=localhost[127.0.0.1]
May 15 10:23:45 mail postfix/cleanup[1235]: 4XYZ123: message-id=<20260515102345.test@opama.dev>
May 15 10:23:45 mail postfix/qmgr[1236]: 4XYZ123: from=<root@opama.dev>, size=456, nrcpt=1
May 15 10:23:46 mail postfix/smtp[1237]: 4XYZ123: to=<test@example.com>, relay=mx.example.com[93.184.216.34]:25, delay=0.5, status=sent (250 OK)

Step 5: Configuring a Mail Client

Once the server is running, point your mail client at mail.opama.dev with these settings:

• IMAP: port 993 with SSL/TLS
• SMTP: port 587 with STARTTLS
• Username: your full email address (anas@opama.dev)
• Password: the system user's password (or whatever you configured for SASL)

Here's what the connection flow looks like:

This diagram shows how mail moves from your client (MUA) through Postfix (MTA) to Dovecot (MDA) and finally to disk.

Security Considerations

Running a mail server comes with responsibilities. Here's a security hardening checklist:

• Strong passwords configured
• Fail2ban installed and running
• SSL certificates set up with auto-renewal
• Rate limits configured in Postfix
• Daily log monitoring scheduled
• SPF, DKIM, and DMARC all verified

Troubleshooting Common Issues

Mail goes to spam

This is almost always a DNS issue. Double-check your SPF, DKIM, and DMARC records. The DKIM Checker and MX Toolbox are invaluable for this.

Can't connect on port 587

Make sure your VPS firewall allows it:

sudo ufw allow 25/tcp
sudo ufw allow 587/tcp
sudo ufw allow 993/tcp
sudo ufw allow 143/tcp

Authentication fails

Verify that Dovecot's auth socket is accessible by Postfix. Check that the socket path in /etc/dovecot/dovecot.conf matches what's expected in /etc/postfix/main.cf.

Conclusion

Setting up your own mail server is one of those projects that teaches you more about the internet in a weekend than a month of reading docs. You'll come out with a deep understanding of DNS, TLS, authentication mechanisms, and the SMTP protocol itself.

The best part? Once it's running, it just works. I've had my setup humming along for years with minimal maintenance.

If you run into issues or have questions, feel free to reach out. And remember -- the first time you send an email from your own server and it arrives in someone's inbox (not spam), it's a genuinely satisfying feeling.