Skip to main content
A mail server on a VPS needs five things before it can deliver reliably: port 25 reachable in both directions, reverse DNS (PTR) for the server IP that matches its hostname such as mail.example.com, an MX record, and SPF, DKIM and DMARC records for your domain. Test port 25 first, then install a maintained mail stack such as Mailcow or Mail-in-a-Box instead of building one by hand.

Before you start

  • Use a fresh server just for mail. Mailcow and Mail-in-a-Box both expect to own ports 25, 80, 443 and the mail ports.
  • Decide whether you really need your own mail server. To only send app email (password resets, order mails), sending through your mailbox provider’s SMTP or a transactional email service is far simpler.
  • You manage spam filtering, updates, backups and your IP’s reputation yourself. A badly configured server can be abused to send spam and get its IP blocklisted.

What does a mail server need to deliver email?

Large mailbox providers check every incoming message. Google’s sender guidelines, for example, require all senders to have SPF or DKIM, valid forward and reverse DNS (PTR) for the sending IP, and TLS. Senders of more than 5,000 messages a day to Gmail must also have SPF and DKIM, plus DMARC.

How to test port 25

Do this before anything else. If port 25 does not work, nothing else matters. Outgoing. Find a mail server to test against, using the MX record of a domain whose mailbox you use (replace example.net):
Then try to connect to the host it prints:
succeeded means outgoing port 25 works. A timeout means something between your server and the internet blocks it. Check your own firewall first; if that is open, open a ticket and ask about outbound SMTP for your service. Incoming. After your mail software is running, test port 25 on your server from another network, as in check if a port is open from outside.
On Ubuntu and Debian, dig is in the dnsutils package and nc in netcat-openbsd. On AlmaLinux / Rocky, install bind-utils and nmap-ncat.

How to set the hostname and reverse DNS

1

Choose the mail hostname

Use a dedicated name such as mail.example.com (Mail-in-a-Box suggests box.example.com). Do not use a name you also host a website on.
2

Create the A record

In your DNS, point mail.example.com to the server IP, for example 203.0.113.10. Add an AAAA record too if the server sends over IPv6.
3

Set reverse DNS

Set the PTR for 203.0.113.10 to mail.example.com. On a VPS, follow configure reverse DNS. On a dedicated server, open a ticket. If you use IPv6, set a PTR for the IPv6 address too.
4

Set the server hostname

See change the hostname.
Check that forward and reverse DNS agree:

Which DNS records do I need?

These are typical records for example.com with one mail server. Your mail software shows the exact DKIM value.
  • SPF v=spf1 mx -all allows only the servers in your MX record to send for the domain. If other services also send as @example.com, add them, or start with ~all (soft fail) while you test.
  • DKIM: the selector (here dkim) depends on your software. Mailcow and Mail-in-a-Box generate the key and show the record.
  • DMARC: start with p=none to receive reports without affecting delivery, then move to p=quarantine or p=reject once SPF and DKIM pass for all your mail.
Building a mail server from Postfix, Dovecot, Rspamd and a webmail client by hand takes many config files, and small mistakes cause lost or rejected mail. A maintained mail stack installs and updates all of these together.

How to install Mailcow

1

Install Docker

Follow install Docker, using Docker’s own repository (Mailcow asks for Docker Engine 24.0.0 or newer).
2

Download Mailcow

3

Generate the config

Enter mail.example.com as the hostname when asked.
4

Start it

5

Log in and change the password

Open https://mail.example.com/admin. The default login is admin / moohoo. Change it at once, add your domain, then copy the DKIM record it shows into your DNS.

How to install Mail-in-a-Box

On a fresh Ubuntu 22.04 server with the hostname box.example.com and reverse DNS already set:
The setup asks for your first email address and the hostname, then installs everything. Then log in to its web control panel, which runs status checks on your DNS records and the system and lists anything to fix.

How to check it worked

1

Check the DNS records

2

Send a test message to a mailbox you own elsewhere

Open the received message, use Show original (or your client’s equivalent) and look for spf=pass, dkim=pass and dmarc=pass.
3

Reply from that mailbox

This tests incoming mail on port 25.

Frequently asked questions

Why do my emails go to spam?

Most often reverse DNS does not match the hostname, SPF or DKIM fail, or the IP has a poor reputation or is on a blocklist. Check the message headers for pass results, confirm the PTR record, and look up your IP on public blocklists.

Can I run a mail server without reverse DNS?

It will run, but many large providers reject or junk mail from IPs without matching forward and reverse DNS. Set the PTR first.

Do I need IPv6 for mail?

No. If your server sends over IPv6, though, the IPv6 address needs its own PTR record and must be included in SPF (the mx mechanism covers it when mail.example.com has an AAAA record).

Can I host email on the same VPS as my website?

You can, but a separate small VPS for mail keeps a web compromise from affecting your mail, and lets Mailcow or Mail-in-a-Box own the ports they need.

VPS and dedicated server guides

Every server guide in one place.

Configure reverse DNS

Set the PTR for your VPS IP.

Point a domain to your server

A, MX and TXT records.

Check if a port is open

Test port 25 from outside.

Install Docker

Needed for Mailcow.

Free control panels

Panels with built-in mail.

Checked against the official Mailcow system, DNS and install documentation, the Mail-in-a-Box setup guide and Google’s email sender guidelines on 4 November 2026; dig and nc -vz syntax checked on Ubuntu 24.04 LTS. The mail stacks were not installed in testing. Commands can differ on other distributions or releases.

Need Extra Help?

If you encounter any issues, our support team is ready to assist:

Save on Your Hosting

Ready to upgrade your hosting? Use code KB20 at checkout for 20% off your first order!

Last Updated: November 2026