SMTP Security Gaps and Open Relay Risks

Despite being one of the oldest internet protocols still in daily use, SMTP has some not-so-simple problems that continue to enable spoofing, spam, and fraud.

SMTP—the Simple Mail Transfer Protocol—was designed in an era when trust was assumed, leaving the door open (literally) for spoofing, spam, and fraud. To understand why DMARC was needed, we first have to look at the holes in SMTP and the open relay era that hackers feasted on.

Why It Matters: SMTP's design flaws continue to be exploited today, enabling phishing, spoofing, and costly business email compromises. Knowing its weaknesses helps explain why DMARC is not just another acronym, but a crucial tool for modern email security.

Learning Objectives

By the end of this guide, you'll understand:

  • The fundamental weaknesses in SMTP that make email spoofing possible
  • How open relays once allowed (and sometimes still allow) spammers to hijack mail servers
  • How these legacy issues led to the development of authentication frameworks like DMARC

SMTP Protocol Architecture and Inherent Weaknesses

SMTP (Simple Mail Transfer Protocol) is defined in RFC 5321 (updated from original RFC 821, 1982). Operating on TCP port 25, SMTP follows a command-response model with no built-in security mechanisms.

SMTP Transaction Flow

# Basic SMTP session S: 220 mail.example.com ESMTP Postfix C: EHLO client.example.org S: 250-mail.example.com Hello client.example.org S: 250-SIZE 52428800 S: 250-8BITMIME S: 250 DSN C: MAIL FROM:<[email protected]> # No identity verification! S: 250 OK C: RCPT TO:<[email protected]> S: 250 OK C: DATA S: 354 Start mail input; end with <CRLF>.<CRLF> C: From: Trusted Person <[email protected]> # Completely forgeable! C: Subject: Urgent Request C: C: Please wire $50,000 immediately. C: . S: 250 OK: queued as 12345 C: QUIT S: 221 Bye

Critical Security Gaps

  • No sender authentication: MAIL FROM command accepts any address
  • Header forgery: From, Reply-To, Return-Path completely spoofable
  • Envelope vs. Header disconnect: Routing uses MAIL FROM, users see From header
  • No message integrity: Content can be modified in transit
  • Plain text protocol: Commands and data transmitted unencrypted
  • Command injection: Insufficient input validation in many implementations

Technical Vulnerability Categories

1. Identity Spoofing

# Attacker spoofs CEO email MAIL FROM:<[email protected]> # Envelope (SPF checks this) From: CEO John Smith <[email protected]> # Header (user sees this) # Result: SPF passes for attacker.com, user trusts [email protected]

2. Command Injection

# SMTP command injection example MAIL FROM:<[email protected]\r\nRCPT TO:<[email protected]>> # Injects additional RCPT TO command # Modern example: SMTP smuggling MAIL FROM:<[email protected]>\r\n.\r\nMAIL FROM:<[email protected]>

3. Header Field Abuse

# Multiple From headers (RFC violation but often processed) From: [email protected] From: [email protected] # Display name spoofing From: "Amazon Security" <[email protected]> # Unicode homograph attacks From: support@аmazon.com # 'а' is Cyrillic, looks identical

Imagine mailing a letter: with SMTP, you can write any return address you want on the envelope. The post office (SMTP servers) will deliver it without question. Now imagine a criminal writing your name and address on threatening letters. That's email spoofing in a nutshell.

Modern SMTP Attack Vectors

1. Protocol-Level Attacks

  • Envelope/Header Mismatch: MAIL FROM vs. From: header discrepancy
  • SMTP Command Injection: Malformed commands to bypass filters
  • SMTP Smuggling: Exploiting parsing differences between servers
  • Connection Hijacking: BGP/DNS attacks to redirect SMTP traffic

2. Authentication Bypass

  • SASL Mechanism Abuse: Weak authentication methods
  • TLS Downgrade: Forcing plaintext communication
  • Certificate Validation Bypass: MITM attacks on SMTP over TLS

3. Advanced Spoofing Techniques

  • Subdomain Hijacking: Taking over abandoned subdomains for spoofing
  • Domain Shadowing: Using compromised DNS to create spoofing subdomains
  • Cousin Domains: Registering similar domains for spoofing
  • Unicode Homographs: Using similar-looking characters from different alphabets

Real-World Attack Example: Business Email Compromise

# Phase 1: Reconnaissance Target: BigCorp Inc (bigcorp.com) CEO: John Smith (LinkedIn, company website) Email pattern: [email protected] # Phase 2: Infrastructure setup Registered domain: bigc0rp.com (0 instead of o) SMTP server: mail.bigc0rp.com SPF record: "v=spf1 a mx ~all" (validates for bigc0rp.com) # Phase 3: Attack execution MAIL FROM:<[email protected]> # Envelope (SPF passes) From: John Smith <[email protected]> # Header (spoofed!) To: [email protected] Subject: Urgent: Confidential Wire Transfer Mary, I'm in a confidential meeting with investors. Please wire $250,000 to the account below. Call my assistant if questions - I'm unavailable. Account: [Attacker's bank details] Regards, John # Result: Finance sees [email protected], SPF passes for bigc0rp.com # Without DMARC alignment check, attack succeeds

Why Was This a Problem?

Because it was cheap and easy to abuse.

In the 1990s, roughly 90% of SMTP servers were open relays. Spammers quickly realized they could send millions of emails from these servers, hiding their real identities. That's why your aunt's dial-up inbox was filled with miracle pills and suspicious bank notices.

Even as spam filters improved, more serious threats emerged:

  • Phishing emails that pass as your boss or your bank
  • Business Email Compromise (BEC), where attackers spoof legitimate domains and trick employees into wiring money
  • Fake invoices, credential harvesting, and malware-laced attachments disguised as internal messages

By the early 2010s, spam made up 85% of all email traffic. And most of it slipped right past basic email defenses.

How Do SMTP Vulnerabilities Actually Work?

Let's break down how attackers exploit the gap between the envelope sender (what servers see) and the header sender (what users see).

Imagine an attacker wants to send an email that appears to come from [email protected]. The attacker controls a different domain, attackers-domain.com, which they have configured with a valid SPF record. Here's what the SMTP conversation looks like:

# The attacker's server connects and introduces itself
EHLO mail.attackers-domain.com

# The attacker uses their *own* domain for the envelope sender
# SPF will check this domain and it will PASS if configured correctly
MAIL FROM:<[email protected]>

# The recipient is the target
RCPT TO:<[email protected]>

# The message data starts
DATA

# The attacker forges the "From" header, which is what the user sees
From: "Company CEO" <[email protected]>
Subject: Urgent: Wire Transfer Request

Please process an urgent wire transfer for $10,000 to our new vendor. Details attached.
.

Here's the critical failure:

  • The receiving mail server performs an SPF check on the MAIL FROM address: [email protected]. Since the attacker controls this domain and its SPF records, the check passes.
  • The victim's email client displays the From: header: "Company CEO" <[email protected]>.

The message appears legitimate, and it passed a key authentication check (SPF), yet it's a complete forgery. This is the exact vulnerability DMARC was designed to close by requiring that the domain in the From: header aligns with the domain in the MAIL FROM (for SPF) or the DKIM signature. Without alignment, a "pass" is meaningless.

Open Relay Vulnerabilities: A Historical Perspective

The Open Relay Era (1980s-2000s)

Open relays were SMTP servers that accepted and forwarded mail from any source to any destination. This was the default configuration for most mail servers until the late 1990s.

Technical Definition

An open relay allows the following transaction from any IP address:

# Testing for open relay telnet mail.victim.com 25 220 mail.victim.com ESMTP HELO spammer.com 250 Hello spammer.com MAIL FROM:<[email protected]> 250 OK RCPT TO:<[email protected]> # External domain! 250 OK # Should reject if not open relay DATA 354 Start mail input From: CEO <[email protected]> Subject: Urgent! Fake message content . 250 Message accepted # Open relay confirmed!

Modern Open Relay Detection

# Automated testing script #!/bin/bash SMTP_SERVER="$1" TEST_FROM="[email protected]" TEST_TO="[email protected]" # Test multiple scenarios for TEST in "basic" "percent" "source-route" "double-bounce"; do echo "Testing $TEST relay method..." case $TEST in "basic") echo -e "HELO test\nMAIL FROM:<$TEST_FROM>\nRCPT TO:<$TEST_TO>\nQUIT" | nc $SMTP_SERVER 25 ;; "percent") echo -e "HELO test\nMAIL FROM:<$TEST_FROM>\nRCPT TO:<$TEST_TO%openrelay.org>\nQUIT" | nc $SMTP_SERVER 25 ;; esac done

SMTP Smuggling: Modern Open Relay Variant

SMTP smuggling exploits differences in how mail servers parse SMTP commands, allowing injection of additional commands:

# SMTP Smuggling Attack Vector # Server A processes \r\n as command separator # Server B requires \r\n.\r\n for message termination DATA From: [email protected] Subject: Regular Email This is normal content.\r\n # Smuggled command (invisible to Server A) .\r\n MAIL FROM:<[email protected]> RCPT TO:<[email protected]> DATA From: CEO <[email protected]> Subject: Transfer Funds Smuggled malicious content .\r\n # End original message .

Vulnerable SMTP Implementations

  • Postfix < 3.8.3: Inconsistent newline handling
  • Sendmail: Command parsing edge cases
  • Microsoft Exchange: Legacy protocol handling
  • Cloud providers: AWS SES, SendGrid filtering bypasses

Statistical Impact

  • 1990s Peak: 90% of SMTP servers were open relays
  • 2000: 80% of spam sent through open relays
  • 2023: ~0.1% of servers still misconfigured as open relays
  • Current threat: SMTP smuggling affects 20+ major email providers

SMTP Server Configuration Analysis

Postfix Configuration (Secure vs. Vulnerable)

Minimal Anti-Relay Configuration

# /etc/postfix/main.cf - Basic anti-relay smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination # Explanation: # permit_mynetworks: Allow relay from trusted IPs # permit_sasl_authenticated: Allow authenticated users # reject_unauth_destination: Block external relay attempts

Comprehensive Security Configuration

# /etc/postfix/main.cf - Hardened configuration # Anti-relay controls smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, reject_unauth_pipelining, reject_invalid_helo_hostname, reject_non_fqdn_helo_hostname, reject_non_fqdn_sender, reject_non_fqdn_recipient, reject_unknown_sender_domain, reject_unknown_recipient_domain # Rate limiting smtpd_client_connection_count_limit = 10 smtpd_client_connection_rate_limit = 30 smtpd_client_message_rate_limit = 100 # Size restrictions message_size_limit = 51200000 mailbox_size_limit = 0 # Header checks header_checks = regexp:/etc/postfix/header_checks body_checks = regexp:/etc/postfix/body_checks # SMTP smuggling protection smtpd_data_restrictions = reject_unauth_pipelining

Header Filtering Rules

# /etc/postfix/header_checks - Anti-spoofing rules # Block multiple From headers /^From:.*\nFrom:/i REJECT Multiple From headers detected # Block suspicious display names /^From:.*"(PayPal|Amazon|Apple|Microsoft)"/i REJECT Suspicious display name # Block Unicode homographs /^From:.*[\x{0430}-\x{044F}]/i REJECT Cyrillic characters in From header # Block common BEC patterns /^Subject:.*(urgent|wire|transfer|payment).*CEO/i HOLD Potential BEC attempt

Sendmail Configuration

# /etc/mail/sendmail.mc - Anti-relay configuration define(`confPRIVACY_FLAGS', `authwarnings,novrfy,noexpn,restrictqrun')dnl FEATURE(`access_db')dnl FEATURE(`blacklist_recipients')dnl FEATURE(`delay_checks')dnl # Access control FEATURE(`relay_based_on_MX')dnl # Rate limiting define(`confCONNECTION_RATE_THROTTLE', `15')dnl define(`confMAX_DAEMON_CHILDREN', `40')dnl

Microsoft Exchange Configuration

# PowerShell commands for Exchange anti-relay # Set receive connector restrictions Set-ReceiveConnector "Internet Connector" -PermissionGroups AnonymousUsers -AuthMechanism None # Configure anti-spam settings Set-TransportConfig -ExternalPostmasterAddress [email protected] Set-SenderReputationConfig -Enabled $true Set-ContentFilterConfig -Enabled $true # Block open relay Set-ReceiveConnector "Internet Connector" -RequireEHLODomain $true

Testing SMTP Security

Manual Testing Commands

# Test 1: Basic open relay test telnet smtp.target.com 25 HELO test.com MAIL FROM:<[email protected]> RCPT TO:<[email protected]> # Should be rejected if properly configured # Test 2: SMTP smuggling detection echo -e "DATA\nFrom: [email protected]\nSubject: Test\n\nNormal message.\r\n.\r\nMAIL FROM:<[email protected]>\nQUIT" | nc smtp.target.com 25 # Test 3: Header injection echo -e "MAIL FROM:<[email protected]>\nRCPT TO:<[email protected]>\nDATA\nFrom: [email protected]\nSubject: Test\nBcc: [email protected]\n\nTest message\n.\nQUIT" | nc smtp.target.com 25

Automated Security Testing Tools

# Nmap SMTP enumeration nmap -p 25 --script smtp-open-relay target.com nmap -p 25 --script smtp-enum-users target.com nmap -p 25 --script smtp-vuln-cve2011-1764 target.com # OpenRelay.org testing curl -X POST https://www.openrelay.org/relay-test \ -d "server=target.com&port=25" # Custom Python SMTP security scanner python3 -c " import smtplib server = smtplib.SMTP('target.com', 25) server.set_debuglevel(1) try: server.mail('[email protected]') server.rcpt('[email protected]') print('VULNERABLE: Open relay detected') except: print('SECURE: Relay rejected') server.quit() "

Common Pitfalls and FAQs

1. "I'm not an open relay anymore. I'm safe, right?"

Not entirely. If your domain lacks SPF, DKIM, or DMARC, attackers can still spoof your email in ways your recipients can't detect.

2. "Why didn't SPF and DKIM solve this?"

SPF checks the IP address that sends mail, but only for the envelope sender—not what the user sees. DKIM signs message headers but doesn't require the signing domain to match the "From" address. Both lacked alignment and enforcement.

3. "Is SMTP still dangerous?"

Yes, if left unguarded. Modern attacks exploit its design flaws in subtle ways (like SMTP smuggling). But when paired with DMARC, SPF, and DKIM, it's much safer.

What Came Next?

Because SMTP couldn't evolve quickly, the security community developed add-ons. SPF and DKIM came first. But without enforcement, spoofers still got through.

That's where DMARC entered.

DMARC checks whether SPF or DKIM passed and whether their domains align with the visible "From" address. It then lets the domain owner say, "If this fails, reject it—or at least quarantine it." It also provides reporting, so you can see who's trying to spoof your domain.

Historical Context: Think of SMTP as a legacy building with no locks on the doors. DMARC is the modern security system bolted on to keep impersonators out—and finally let us trust email again.

Next Steps

To dive deeper into how DMARC solves these issues, check out:

Understanding SMTP's weaknesses is the first step in appreciating why modern email authentication protocols were developed. In the next sections, we'll explore how SPF and DKIM attempted to address these issues, and why DMARC was ultimately needed to tie it all together.