Secure SMTP Server Configuration: Ports, TLS and the Auth That Stops Spoofing

Secure SMTP Server Configuration: Ports, TLS and the Auth That Stops Spoofing

Updated:

Secure SMTP configuration comes down to three decisions. You choose which port your clients submit on, whether encryption is mandatory or optional, and whether the server demands a login before it accepts a message. Get those three right and spoofing through your mail server becomes very hard. Get them wrong and you have built an open relay with a friendly welcome mat.

Most home users and small businesses never touch this. They paste a hostname and a password into a mail app and hope. That works until a spammer finds the server, or until Gmail starts rejecting your outgoing mail because your domain looks forged. The settings below are the ones that actually matter, in the order I would fix them.

Start With the Ports, Because They Decide Everything Else

The port tells the server what kind of conversation to expect before a single byte of mail is sent. Mixing them up is the most common reason a client “just won’t connect.”

Servers talk to each other on TCP port 25. That is the port-to-port channel between mail servers, and it is the one you should never open to your users or expose casually to the internet. User-level email clients submit outgoing mail on port 465 or 587 per RFC 8314. That split is the whole game.

The standard submission port usually starts unencrypted and upgrades with STARTTLS. The other submission port is used with encryption from the first byte, which is why people call it implicit TLS. The RFC favors implicit TLS on separate SSL ports over STARTTLS for email protocols.

So my default recommendation for a new setup is submission on the standard port with STARTTLS required, or the implicit TLS port if your provider supports it cleanly. Server-to-server delivery stays on its own port.

If you are still hunting for the right hostname and port for a specific provider, this walkthrough on SMTP settings explained and where to find yours covers the lookup side well.

Mail client account settings showing SMTP server and port fields
The port field is where most failed setups begin, and where the security decisions get made.

Turn On SMTP-AUTH and Stop Being an Open Relay

An open relay is a mail server that forwards mail for anyone who asks. In 1998 the Internet Mail Consortium reported that 55% of mail servers were open relays. By 2002 that figure had dropped to less than 1%. The industry closed this hole because spammers burned it down.

The tool that closed it is SMTP-AUTH, defined in RFC 4954, which gives the server an access control mechanism for mail clients. In plain terms, the server asks for a username and password before it will accept a message for delivery. No login, no relay.

Here is what that looks like in practice on a typical mail server:

  1. Configure SMTP-AUTH according to your server’s submission and relay policy.
  2. Require authentication for all outgoing mail from your users.
  3. Restrict relaying so only authenticated users can send to external domains.
  4. Use a full email address as the login name, since most servers key accounts that way.
  5. Store passwords hashed, and force a change if an account is compromised.
  6. Log authentication failures and watch for repeated attempts from one IP.
  7. Test from an outside network to confirm an unauthenticated client is refused.
Turn On SMTP-AUTH and Stop Being an Open Relay
1Enable SMTP-AUTH
2Require authentication
3Restrict relaying so only authenticated users
4Use a full email address as the login name
5Store passwords hashed
6Log authentication failures
7Test from an outside network
The steps from this section at a glance.

That last step matters. Plenty of people test from inside the LAN, where the server trusts the local subnet, and conclude everything works. Then they travel and nothing sends.

Force TLS Instead of Offering It

STARTTLS for SMTP is defined in RFC 3207. It is a command the client sends to upgrade a plain connection into an encrypted one. The catch is that it is optional by design, and an attacker sitting in the middle can strip the upgrade request so the conversation stays in cleartext.

That is why the modern guidance leans toward implicit TLS. If your server and clients both support implicit TLS from the first byte, use it. There is no downgrade window.

Whichever you pick, set the server to require encryption for authenticated submission. A client that cannot do TLS should fail loudly. It should never send a password in the clear. On the receiving side, RFC 8460, SMTP TLS Reporting, describes a mechanism for sharing statistics about TLS failures with recipient domains.

For a small setup, the practical checklist is short. Get a certificate from a real certificate authority, renew it automatically, and confirm the mail app is set to “SSL/TLS” on the implicit TLS port or “STARTTLS” on the standard submission port. Those two labels are not interchangeable.

Small office server rack running a mail server
A mail server in a closet still faces the same port and TLS decisions as a data centre.

Match the Auth to the Spoofing Problem

Authentication stops strangers from relaying through your server. It does not, by itself, stop someone forging your domain in the From line and sending from their own machine. That is a separate problem, and it is the one that fills spam folders with fake invoices from your address.

The fix is at the domain level. Publish records that tell receiving servers which machines are allowed to send for your domain, and sign your outgoing mail so tampering is visible. Your provider may document these settings in its control panel. Check the label and your local rules for the exact record names, because they differ between hosts.

There is a second layer people forget. Each SMTP hop is a formal handoff of responsibility for the message, per RFC 5321 section 2.1. That means the server that accepts your message owns it from that point. If your submission server is misconfigured, the next hop inherits the mess. Clean authentication at submission is what keeps that handoff honest.

If you are running this on cheap shared hosting, understand what you are getting. A free host may give you a mail account but throttle or block outbound SMTP entirely to protect its IP reputation. My breakdown of what free hosting really costs you covers where those limits usually bite.

The Settings Most People Get Wrong

A short list of the mistakes I see repeatedly, and what to do instead.

Using the server-to-server port for client submission. It is generally discouraged; that port is meant for traffic between mail servers. Many providers block it outright for residential connections.

Leaving STARTTLS optional. Optional means strippable. Require it, or use implicit TLS on 465.

Sharing one mailbox login across devices and staff. When one device is compromised you have to rotate the password everywhere. Use separate accounts.

Skipping the reverse DNS and domain records. Receiving servers weigh these heavily. A perfectly configured submission port still lands in spam if your domain looks unauthenticated.

Testing only inside the network. Always confirm from an outside connection that unauthenticated relay is refused.

Ignoring the logs. Failed auth attempts from foreign IPs are the early warning that someone is probing your server.

When to Call a Pro

If you run a single mailbox for yourself, the settings above are manageable in an afternoon. If you run mail for a team, a business domain, or anything that sends transactional email, the calculus changes.

Call in help when you are responsible for deliverability across a domain, when you need to prove to a client that their mail is authenticated, or when your server has been flagged for sending spam. Cleaning up a blacklisted IP and rebuilding trust takes expertise and patience, and guessing makes it worse.

The same applies to anything touching file transfer alongside mail. If you are moving files between servers and mail, the underlying protocol questions overlap more than people expect, and a plain-English primer on what FTP stands for is a reasonable place to start untangling them.

Home office user reviewing mail server logs with a consultant
Bringing in help early is cheaper than rebuilding a domain's reputation after a spam listing.

Secure SMTP FAQ

Which port should I use for sending mail from a mail app?

Use the STARTTLS submission port, or the implicit TLS port. Both are the standard submission ports for user-level email clients per RFC 8314. Server-to-server delivery runs on a separate port from client submission.

What does SMTP-AUTH actually protect against?

SMTP-AUTH requires a username and password before the server accepts a message for delivery, which stops strangers from using your server as an open relay. It is defined in RFC 4954 as an access control mechanism for mail clients. It does not, on its own, stop someone forging your domain from their own server.

Is STARTTLS secure enough on its own?

STARTTLS can be stripped by an attacker in the middle of the connection, which downgrades the session to cleartext. The same RFC that defines the submission ports favors implicit TLS on separate SSL ports for that reason. If you must use STARTTLS, configure the server to require it so the upgrade cannot be skipped.

Why does my mail server still get flagged for spam if I use authentication?

Authentication controls who can relay, while domain-level records control whether receiving servers trust your From address. You need both. A correctly authenticated submission port still lands in spam if your domain looks unauthenticated to the receiving server.

How do I know my TLS is actually working?

Send a test message to an external address and inspect the received headers for the encryption details, or use your provider’s TLS reporting if it offers one. RFC 8460 describes a reporting mechanism for sharing TLS failure statistics with recipient domains. If the headers show no encryption, your client is falling back to plaintext.

Thank you for reading!

Roosevelt Naylor
Roosevelt Naylor
Editor

Roosevelt Naylor writes about referral programmes and word-of-mouth growth on Limitless Referrals: what actually makes someone recommend a business, and why most reward schemes never manage it. He prefers plain arithmetic to growth-hacking claims.

Leave a Comment

Your email address will not be published. Required fields are marked *

As an Amazon Associate we earn from qualifying purchases.