Skip to main content
← Blog
GuidesVPSmail serverPostfixDovecotDKIMemail hostingLinuxself-hosted

How to Install & Configure a Mail Server on Your VPS (Postfix + Dovecot + DKIM)

Step-by-step guide to setting up a production-ready mail server on a FlashRDP Linux VPS with Postfix, Dovecot, DKIM, SPF, DMARC, TLS encryption, and optional Roundcube webmail.

Rohan
Rohan
13 min read
How to Install & Configure a Mail Server on Your VPS (Postfix + Dovecot + DKIM)

Running your own mail server gives you complete control over your email infrastructure, from custom domains and unlimited mailboxes to full privacy without relying on third-party providers like Gmail or Outlook. This guide walks you through installing and configuring a production-ready mail server on a FlashRDP Linux VPS using Postfix (SMTP), Dovecot (IMAP/POP3), and essential security layers like SPF, DKIM, and DMARC.

Last updated: May 8, 2026

Self-hosting email is one of the most rewarding (and educational) projects you can run on a VPS. According to Radicati Group's 2025 Email Statistics Report, over 376 billion emails are sent daily worldwide, and while most people rely on hosted email services, the privacy-conscious and technically minded increasingly choose self-hosted solutions.

Prerequisites

Before you begin, make sure you have:

  • A FlashRDP Linux VPS with at least 2 vCPU, 2 GB RAM, and Ubuntu 22.04/24.04 or Debian 12
  • A registered domain name (e.g., yourdomain.com)
  • Root (sudo) access to your VPS
  • A clean IP address (not on any email blacklists)
  • DNS access to configure MX, A, SPF, DKIM, and DMARC records
Tip

💡 Tip FlashRDP Linux VPS plans include full root access, 1 Gbps unmetered bandwidth, and clean IPs suitable for mail server hosting. Plans start at $11.99/month. View Linux VPS plans.

Step 1: Initial Server Setup

1.1 Update and Secure Your VPS

Connect to your server via SSH and run the initial setup:

×
-
+
bash
# Update system packages
sudo apt update && sudo apt upgrade -y

# Set the hostname to your mail domain
sudo hostnamectl set-hostname mail.yourdomain.com

# Update /etc/hosts
echo "YOUR_SERVER_IP mail.yourdomain.com mail" | sudo tee -a /etc/hosts

# Install essential tools
sudo apt install -y curl wget gnupg2 software-properties-common ufw

1.2 Configure the Firewall

Open only the ports needed for email and SSH:

×
-
+
bash
sudo ufw default deny incoming
sudo ufw default allow outgoing

# SSH
sudo ufw allow 22/tcp

# SMTP
sudo ufw allow 25/tcp

# SMTP Submission
sudo ufw allow 587/tcp

# SMTPS (Implicit TLS)
sudo ufw allow 465/tcp

# IMAP
sudo ufw allow 143/tcp

# IMAPS
sudo ufw allow 993/tcp

# HTTP/HTTPS (for Let's Encrypt and optional webmail)
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp

sudo ufw enable

Step 2: Configure DNS Records

Before installing any mail software, configure your DNS records with your domain registrar. These are critical for mail delivery and authentication.

2.1 Essential DNS Records

Record TypeNameValueTTL
Amail.yourdomain.comYOUR_SERVER_IP300
MXyourdomain.commail.yourdomain.com (Priority: 10)300
SPF (TXT)yourdomain.comv=spf1 mx a ip4:YOUR_SERVER_IP -all300
DMARC (TXT)_dmarc.yourdomain.comv=DMARC1; p=quarantine; rua=mailto:postmaster@yourdomain.com300
PTR (rDNS)YOUR_SERVER_IPmail.yourdomain.com300

ℹ️ Note The PTR (reverse DNS) record can be configured directly from your FlashRDP service area. Navigate to Service Area > Network tab and set your rDNS hostname to mail.yourdomain.com. Changes typically propagate within a few minutes.

2.2 Verify DNS Propagation

×
-
+
bash
# Check MX record
dig +short MX yourdomain.com

# Check A record
dig +short A mail.yourdomain.com

# Check SPF record
dig +short TXT yourdomain.com

Step 3: Install SSL/TLS Certificate

A valid TLS certificate is essential for encrypted email transmission. We will use Let's Encrypt for free certificates.

×
-
+
bash
# Install Certbot
sudo apt install -y certbot

# Obtain a certificate for the mail hostname
sudo certbot certonly --standalone -d mail.yourdomain.com

# Verify the certificate was created
sudo ls -la /etc/letsencrypt/live/mail.yourdomain.com/

Set up automatic renewal:

×
-
+
bash
# Test renewal
sudo certbot renew --dry-run

# Certbot installs a systemd timer by default
sudo systemctl status certbot.timer

Step 4: Install and Configure Postfix (SMTP)

Postfix handles outgoing and incoming email delivery (SMTP).

4.1 Install Postfix

×
-
+
bash
sudo apt install -y postfix

# During installation, select:
# - General type of mail configuration: Internet Site
# - System mail name: yourdomain.com

4.2 Configure Postfix Main Settings

Edit the main configuration file:

×
-
+
bash
sudo nano /etc/postfix/main.cf

Replace/merge with the following configuration:

×
-
+
ini
# Basic settings
myhostname = mail.yourdomain.com
mydomain = yourdomain.com
myorigin = $mydomain
mydestination = $myhostname, $mydomain, localhost.$mydomain, localhost
mynetworks = 127.0.0.0/8 [::1]/128

# TLS settings
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
smtpd_tls_security_level = may
smtpd_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1
smtp_tls_security_level = may
smtp_tls_protocols = !SSLv2, !SSLv3, !TLSv1, !TLSv1.1

# SASL Authentication (for Dovecot)
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
smtpd_sasl_auth_enable = yes
smtpd_sasl_security_options = noanonymous
smtpd_sasl_local_domain = $myhostname

# Restrictions
smtpd_recipient_restrictions =
    permit_sasl_authenticated,
    permit_mynetworks,
    reject_unauth_destination,
    reject_invalid_hostname,
    reject_non_fqdn_hostname,
    reject_non_fqdn_sender,
    reject_non_fqdn_recipient,
    reject_unknown_sender_domain,
    reject_rbl_client zen.spamhaus.org,
    reject_rbl_client bl.spamcop.net

# Mailbox delivery (Maildir format via Dovecot)
home_mailbox = Maildir/
mailbox_size_limit = 0
recipient_delimiter = +

# Connection limits
smtpd_client_connection_rate_limit = 30
smtpd_client_message_rate_limit = 60
message_size_limit = 52428800

4.3 Configure Submission Port (587)

Edit the master configuration:

×
-
+
bash
sudo nano /etc/postfix/master.cf

Uncomment/add the submission block:

×
-
+
ini
submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_tls_auth_only=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_reject_unlisted_recipient=no
  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
  -o milter_macro_daemon_name=ORIGINATING

4.4 Restart Postfix

×
-
+
bash
sudo systemctl restart postfix
sudo systemctl enable postfix
sudo systemctl status postfix

Step 5: Install and Configure Dovecot (IMAP/POP3)

Dovecot handles mailbox access, letting you read email from clients like Thunderbird, Outlook, or Apple Mail.

5.1 Install Dovecot

×
-
+
bash
sudo apt install -y dovecot-core dovecot-imapd dovecot-lmtpd dovecot-pop3d

5.2 Configure Dovecot Mail Location

×
-
+
bash
sudo nano /etc/dovecot/conf.d/10-mail.conf

Set the mail location to Maildir format:

×
-
+
ini
mail_location = maildir:~/Maildir

5.3 Configure Authentication

×
-
+
bash
sudo nano /etc/dovecot/conf.d/10-auth.conf
×
-
+
ini
disable_plaintext_auth = yes
auth_mechanisms = plain login

5.4 Configure TLS

×
-
+
bash
sudo nano /etc/dovecot/conf.d/10-ssl.conf
×
-
+
ini
ssl = required
ssl_cert = </etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
ssl_key = </etc/letsencrypt/live/mail.yourdomain.com/privkey.pem
ssl_min_protocol = TLSv1.2
ssl_prefer_server_ciphers = yes

5.5 Configure Dovecot for Postfix SASL

×
-
+
bash
sudo nano /etc/dovecot/conf.d/10-master.conf

Find the service auth section and configure it:

×
-
+
ini
service auth {
  unix_listener /var/spool/postfix/private/auth {
    mode = 0660
    user = postfix
    group = postfix
  }
}

5.6 Restart Dovecot

×
-
+
bash
sudo systemctl restart dovecot
sudo systemctl enable dovecot
sudo systemctl status dovecot

Step 6: Create Email Accounts

Create system users for each email address. Each system user corresponds to one mailbox.

×
-
+
bash
# Create a mail user (e.g., info@yourdomain.com)
sudo adduser info --disabled-login --gecos "Info Mailbox"

# Set a password for IMAP/SMTP login
sudo passwd info

# Create the Maildir structure
sudo -u info mkdir -p /home/info/Maildir/{cur,new,tmp}

Repeat for additional mailboxes:

×
-
+
bash
sudo adduser support --disabled-login --gecos "Support Mailbox"
sudo passwd support
sudo -u support mkdir -p /home/support/Maildir/{cur,new,tmp}

Step 7: Configure DKIM (DomainKeys Identified Mail)

DKIM adds a cryptographic signature to every outgoing email, proving it came from your server and was not tampered with in transit.

7.1 Install OpenDKIM

×
-
+
bash
sudo apt install -y opendkim opendkim-tools

7.2 Configure OpenDKIM

×
-
+
bash
sudo nano /etc/opendkim.conf

Add/modify these settings:

×
-
+
ini
AutoRestart             Yes
AutoRestartRate         10/1h
Syslog                  Yes
SyslogSuccess           Yes
LogWhy                  Yes

Canonicalization        relaxed/simple
ExternalIgnoreList      refile:/etc/opendkim/TrustedHosts
InternalHosts           refile:/etc/opendkim/TrustedHosts
KeyTable                refile:/etc/opendkim/KeyTable
SigningTable            refile:/etc/opendkim/SigningTable

Mode                    sv
PidFile                 /var/run/opendkim/opendkim.pid
SignatureAlgorithm      rsa-sha256
UserID                  opendkim:opendkim

Socket                  inet:12301@localhost

7.3 Create DKIM Keys

×
-
+
bash
# Create DKIM directory structure
sudo mkdir -p /etc/opendkim/keys/yourdomain.com

# Generate the key pair
sudo opendkim-genkey -b 2048 -d yourdomain.com -D /etc/opendkim/keys/yourdomain.com -s mail -v

# Set permissions
sudo chown -R opendkim:opendkim /etc/opendkim
sudo chmod 600 /etc/opendkim/keys/yourdomain.com/mail.private

7.4 Configure DKIM Tables

×
-
+
bash
# Create TrustedHosts
sudo tee /etc/opendkim/TrustedHosts > /dev/null <<EOF
127.0.0.1
localhost
mail.yourdomain.com
.yourdomain.com
EOF

# Create KeyTable
sudo tee /etc/opendkim/KeyTable > /dev/null <<EOF
mail._domainkey.yourdomain.com yourdomain.com:mail:/etc/opendkim/keys/yourdomain.com/mail.private
EOF

# Create SigningTable
sudo tee /etc/opendkim/SigningTable > /dev/null <<EOF
*@yourdomain.com mail._domainkey.yourdomain.com
EOF

7.5 Get the DKIM DNS Record

×
-
+
bash
sudo cat /etc/opendkim/keys/yourdomain.com/mail.txt

Copy the output and add it as a TXT record in your DNS:

Record TypeNameValue
TXTmail._domainkey.yourdomain.com(paste the full DKIM key from the command output)

7.6 Connect OpenDKIM to Postfix

Add the following to /etc/postfix/main.cf:

×
-
+
bash
sudo tee -a /etc/postfix/main.cf > /dev/null <<EOF

# DKIM
milter_protocol = 6
milter_default_action = accept
smtpd_milters = inet:localhost:12301
non_smtpd_milters = $smtpd_milters
EOF

Restart both services:

×
-
+
bash
sudo systemctl restart opendkim
sudo systemctl restart postfix

Step 8: Test Your Mail Server

8.1 Send a Test Email

×
-
+
bash
# Install mailutils
sudo apt install -y mailutils

# Send a test email
echo "This is a test from my FlashRDP mail server." | mail -s "Test Email" your-external-email@gmail.com

8.2 Check Mail Logs

×
-
+
bash
# View real-time mail logs
sudo tail -f /var/log/mail.log

# Check for errors
sudo grep -i error /var/log/mail.log | tail -20

8.3 Verify DKIM and SPF

After receiving the test email, check the headers in Gmail:

  1. Open the email in Gmail
  2. Click the three dots menu and select "Show original"
  3. Look for SPF: PASS, DKIM: PASS, and DMARC: PASS

8.4 Use Online Testing Tools

Step 9: Install Webmail (Optional)

For browser-based email access, install Roundcube:

×
-
+
bash
# Install dependencies
sudo apt install -y apache2 php php-mysql php-xml php-mbstring php-intl php-zip php-gd mariadb-server

# Secure MariaDB
sudo mysql_secure_installation

# Create Roundcube database
sudo mysql -u root -p <<EOF
CREATE DATABASE roundcube;
GRANT ALL PRIVILEGES ON roundcube.* TO 'roundcube'@'localhost' IDENTIFIED BY 'STRONG_PASSWORD_HERE';
FLUSH PRIVILEGES;
EOF

# Download and install Roundcube
cd /tmp
wget https://github.com/roundcube/roundcubemail/releases/download/1.6.9/roundcubemail-1.6.9-complete.tar.gz
tar -xzf roundcubemail-1.6.9-complete.tar.gz
sudo mv roundcubemail-1.6.9 /var/www/roundcube
sudo chown -R www-data:www-data /var/www/roundcube

Configure Apache for Roundcube:

×
-
+
bash
sudo tee /etc/apache2/sites-available/roundcube.conf > /dev/null <<EOF
<VirtualHost *:443>
    ServerName mail.yourdomain.com
    DocumentRoot /var/www/roundcube

    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/mail.yourdomain.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/mail.yourdomain.com/privkey.pem

    <Directory /var/www/roundcube>
        Options -Indexes
        AllowOverride All
        Require all granted
    </Directory>
</VirtualHost>
EOF

sudo a2ensite roundcube.conf
sudo a2enmod ssl rewrite
sudo systemctl restart apache2

Then visit https://mail.yourdomain.com and follow the Roundcube installer.

Step 10: Harden and Maintain Your Mail Server

10.1 Install Fail2Ban

Protect against brute-force attacks on your mail server:

×
-
+
bash
sudo apt install -y fail2ban

sudo tee /etc/fail2ban/jail.local > /dev/null <<EOF
[postfix]
enabled = true
port = smtp,465,submission
filter = postfix
logpath = /var/log/mail.log
maxretry = 5
bantime = 3600

[dovecot]
enabled = true
port = pop3,pop3s,imap,imaps
filter = dovecot
logpath = /var/log/mail.log
maxretry = 5
bantime = 3600
EOF

sudo systemctl restart fail2ban
sudo systemctl enable fail2ban

10.2 Set Up Log Rotation

×
-
+
bash
# Mail logs are typically rotated by default via /etc/logrotate.d/rsyslog
# Verify it exists:
cat /etc/logrotate.d/rsyslog

10.3 Monitor Blacklists

Regularly check if your server IP is on any email blacklists:

×
-
+
bash
# Quick check via MXToolbox
curl -s "https://mxtoolbox.com/SuperTool.aspx?action=blacklist:YOUR_SERVER_IP"

Or visit mxtoolbox.com/blacklists.aspx and enter your server IP.

10.4 Automatic Certificate Renewal

Ensure Let's Encrypt certificates renew and services reload automatically:

×
-
+
bash
sudo tee /etc/letsencrypt/renewal-hooks/deploy/mail-restart.sh > /dev/null <<'EOF'
#!/bin/bash
systemctl restart postfix
systemctl restart dovecot
EOF

sudo chmod +x /etc/letsencrypt/renewal-hooks/deploy/mail-restart.sh

Mail Server Architecture Overview

Here is how the components work together:

ComponentRolePort(s)
PostfixSMTP server (sends/receives mail)25, 587, 465
DovecotIMAP/POP3 server (mailbox access)143, 993, 110, 995
OpenDKIMSigns outgoing mail with DKIM12301 (internal)
Let's EncryptTLS certificatesN/A
Fail2BanBrute-force protectionN/A
RoundcubeWebmail interface (optional)443

Troubleshooting Common Issues

Emails Going to Spam

  • Check SPF, DKIM, and DMARC records are all configured and passing
  • Verify rDNS (PTR record) matches your mail hostname
  • Test with mail-tester.com for a detailed deliverability score
  • Warm up your IP by sending small volumes initially and gradually increasing

Connection Refused on Port 25

  • Check if your hosting provider blocks outbound port 25 (some do by default)
  • Contact FlashRDP support if needed (port 25 is open by default on all plans)
  • Verify Postfix is running: sudo systemctl status postfix

Authentication Failures

  • Check Dovecot logs: sudo tail -f /var/log/mail.log | grep dovecot
  • Verify the user password is correct: sudo doveadm auth test username password
  • Ensure the SASL socket path matches between Postfix and Dovecot

Certificate Errors

  • Verify certificate files exist: sudo ls -la /etc/letsencrypt/live/mail.yourdomain.com/
  • Check certificate expiry: sudo certbot certificates
  • Renew if needed: sudo certbot renew

Why FlashRDP for Mail Server Hosting

FlashRDP VPS plans are purpose-built for self-hosted services like mail servers:

  • Clean IP Addresses: Our IPs are not on major email blacklists
  • Port 25 Open by Default: No need to request SMTP port unblocking
  • Self-Service rDNS: Configure reverse DNS directly from your Service Area > Network tab without needing to open a support ticket
  • Additional IPs: Need dedicated IPs for different mail domains? Additional IPv4 addresses are available as an addon at cost
  • KVM Virtualization: True hardware isolation ensures consistent performance
  • NVMe Storage: Fast I/O for mail spool operations
  • Unmetered 1 Gbps Bandwidth: Handle high email volumes without overage charges
  • DDoS Protection: Always-on Layer 4 mitigation keeps your mail server online
  • No KYC Required: Deploy with 50+ cryptocurrencies. No identity documents needed.

Plans start at $11.99/month. View all VPS plans.

Frequently Asked Questions

Can I run a mail server on a VPS?

Yes, absolutely. A VPS with at least 2 GB RAM and a clean IP address is sufficient for a personal or small business mail server. FlashRDP VPS plans include everything you need: full root access, clean IPs, open port 25, and self-service rDNS configuration.

Is self-hosting email worth it?

For privacy, control, and learning, yes. Self-hosted email gives you zero dependence on third parties, full data ownership, and the ability to create unlimited mailboxes. However, it requires ongoing maintenance (updates, monitoring, backups). For most businesses, a hybrid approach works best: self-host for privacy-critical mail, use a relay for marketing emails.

How do I prevent my emails from going to spam?

Configure all three authentication records (SPF, DKIM, DMARC), set up a matching rDNS/PTR record (configurable from your FlashRDP Service Area under the Network tab), use a clean IP, warm up your sending reputation gradually, and test with tools like mail-tester.com. Following this guide sets up all these layers.

How much resources does a mail server need?

A basic mail server for under 50 users needs about 1-2 GB RAM and 1-2 vCPU. For larger setups with spam filtering (SpamAssassin or Rspamd), webmail, and virus scanning (ClamAV), plan for 4+ GB RAM.

Can I use this setup for bulk email or newsletters?

This guide is for transactional and personal email, not mass marketing. For bulk email, consider using a dedicated email relay service (Amazon SES, Mailgun, Postmark) alongside your mail server. Using your primary mail server for bulk sends risks blacklisting your IP.

Do I need a static IP for a mail server?

Yes. All FlashRDP VPS plans include a dedicated static IPv4 address, which is essential for consistent email delivery and proper rDNS configuration. If you need additional IPs for multiple domains, these can be added as an addon.

How do I configure rDNS/PTR on FlashRDP?

Log in to your FlashRDP client area, navigate to your VPS service, and go to the Network tab. You can set your reverse DNS hostname directly from there without needing to contact support.

Rohan

Rohan

Operations Manager & Founder

Operations Manager at FlashRDP. With 5+ years in cloud infrastructure, Rohan specializes in KVM virtualization, network security, and building privacy-focused hosting solutions for professionals worldwide.

Related Articles

Ultimate Guide to Hosting SEO Crawlers on a VPS
Mar 20, 2026

Ultimate Guide to Hosting SEO Crawlers on a VPS

Read Article
Ultimate MT4 VPS Setup Guide for Forex Traders
Mar 20, 2026

Ultimate MT4 VPS Setup Guide for Forex Traders

Read Article
Windows RDP Security: 5 Essential Best Practices to Stop Hackers
Mar 20, 2026

Windows RDP Security: 5 Essential Best Practices to Stop Hackers

Read Article