Stopping Spam at the Door: Protocol-Level Email Security

Most businesses assume spam filtering happens after an email arrives — a message lands in the inbox, gets scanned, then either stays put or is moved to junk.

In reality, well-run mail servers do a lot of work before they accept a message at all. This is known as protocol-level or connection-level protection: rules that identify suspicious senders during the SMTP handshake and reject them early.

Outgoing identity vs incoming protection

Many organisations have heard of SPF, DKIM and DMARC. Those standards primarily protect outgoing email identity: they help other mail providers verify that email claiming to come from your domain is authentic.

That matters for stopping spoofing, improving deliverability, and keeping your brand trustworthy — but it does not, by itself, stop your server being probed and abused by the constant background noise of the internet.

The constant background noise of SMTP

If you run a public mail server, you quickly learn a surprising fact: your SMTP port is constantly hit by automated systems. Some are legitimate (Google, Microsoft, Proton), but many are not.

• Compromised websites attempting to send spam
• Botnets scanning for misconfiguration or relay opportunities
• Cheap bulk-mailing scripts that do not behave like real mail servers
• Misconfigured clients trying to send mail directly from residential networks

One of the most useful exercises is simply analysing your mail logs: it becomes obvious how much traffic has nothing to do with normal business email.

Stop spam before it’s accepted

The goal of protocol-level protection is not “perfect filtering”. It’s efficiency and damage reduction: block the obvious junk as early as possible, using lightweight checks that cost you very little.

When a message is rejected before acceptance:

• it never enters your mail queue
• your content filters don’t waste CPU scanning it
• it never reaches end users or mailboxes
• attackers get less feedback and less opportunity

Reverse DNS validation

A simple, powerful signal is whether the sending IP has valid reverse DNS (PTR) and it matches expected hostname patterns. Legitimate mail systems almost always have this configured. Many spam sources do not.

HELO and sender domain sanity checks

Two more early-stage checks are:

• HELO/EHLO validation: does the client announce a plausible hostname, in a valid format, that actually exists?
• Sender domain validation: is the domain part of the MAIL FROM address real and resolvable?

These are not magic bullets — determined spammers can pass them — but they are excellent at filtering out sloppy tooling, scanners, and compromised systems with broken configurations.

Greylisting with Postgrey

Greylisting is one of those techniques that sounds almost too simple — but it remains extremely effective against compromised-site spam.

The principle: when your server sees a new sender combination, it temporarily rejects the first delivery attempt with a 4xx response (e.g. 450). A properly configured mail server will retry after a short delay. Most spam scripts won’t.

Postgrey tracks a “triplet”:

• client IP address
• sender address
• recipient address

Large providers (Google, Microsoft, Proton) are often whitelisted by default because greylisting them causes unnecessary initial delays for legitimate mail. The technique is primarily aimed at the lower-quality sources: compromised websites and one-shot spam tools.

Trade-offs and what to expect

No security control is free of trade-offs. The practical impacts to understand are:

• Initial delay for some legitimate senders: when a new sender is greylisted, delivery may be delayed by a few minutes (depending on their retry schedule).
• Some checks can affect misconfigured clients: for example, residential networks often lack reverse DNS, so direct-to-server sending from phones/laptops may be rejected (as it should be).
• Whitelisting matters: reputable senders can be exempted where delays would be counterproductive.

The goal is not to make email “hard to use”. It’s to make abusive traffic expensive and inconvenient — while keeping normal business email smooth.

Practical steps for SMEs

If you run (or are considering) self-hosted email, a sensible security baseline includes:

• Outbound authentication: SPF, DKIM and DMARC configured correctly
• MFA for mail accounts: reduce damage if credentials are stolen
• Protocol-level filtering: reverse DNS checks, HELO/sender sanity checks, and greylisting
• Log visibility: basic reporting so you can see what is hitting your server
• Clear escalation path: when an incident involves impersonation or client risk