Configuring SMTP Relay in Exchange Server SE & Exchange Online: Anonymous vs Authenticated & High Volume Email (HVE)

In every enterprise IT infrastructure, line-of-business (LOB) applications, multi-function printers (MFPs), database monitoring alerts, and backup appliances require an SMTP gateway to dispatch automated notifications. Historically, systems administrators configured "anonymous relay" on on-premises Exchange servers.

However, poorly configured anonymous relay connectors represent a severe security vulnerability. If a connector is accidentally exposed to external subnets or lacks strict IP whitelisting, malicious actors can turn your Exchange server into an open mail relay, resulting in immediate domain blacklisting across Spamhaus, Barracuda, and Microsoft Defender. Furthermore, with Microsoft 365 enforcing modern authentication and throttling SMTP AUTH, cloud-bound organizations need modern alternatives like High Volume Email (HVE).

In this modernized guide, we examine how to configure a secure, tightly scoped internal anonymous relay connector on Exchange Server SE and 2019, and contrast it with cloud options in Exchange Online.


Relay Architectures: On-Premises Scoped vs. Cloud Options

Depending on your mail topology, select the appropriate relay pattern:

  1. Exchange On-Premises Scoped Receive Connector: Ideal for legacy devices inside the corporate network that lack TLS 1.3 or authentication capabilities. Traffic is restricted by explicit internal IP addresses, and external routing permissions are granted only to those trusted IPs.
  2. Exchange Online High Volume Email (HVE): A cloud-native service specifically designed for massive internal transactional and alert broadcasts without hitting standard tenant-wide recipient rate limits.
  3. Exchange Online Inbound Connector (Direct Send): Authenticates devices using your static public egress IP address or a dedicated TLS certificate.
SMTP Relay Decision Architecture: On-Premises SE vs. Exchange Online Option A: On-Premises Scoped Connector (SE / 2019) • Best For: - Legacy copiers, SCADA systems, internal printers • Key Configuration: - Role: FrontendTransport (TCP 25) - RemoteIPRanges: Strict static appliance IPs - Extended Right: ms-Exch-SMTP-Accept-Any-Recipient ✔ Zero Open Relay risk; non-routable subnets only Option B: Cloud High Volume Email (HVE) • Best For: - Cloud apps, Azure VMs, high-volume marketing/reports • Key Configuration: - Endpoint: smtp-hve.office365.com (Port 587 TLS) - Dedicated HVE Service Account (No M365 license needed) - Daily threshold: Up to 100,000 internal recipients ✔ Eliminates tenant-wide transport throttling Golden Rules for Internal Relay Security: 1. NEVER add 0.0.0.0-255.255.255.255 or an entire /16 subnet to an anonymous relay connector. 2. Ensure your SPF record includes all external public NAT IPs used by relaying devices to prevent DMARC rejection.

Step-by-Step: Creating a Scoped Relay Connector in Exchange Server SE / 2019

1. Create the Custom Frontend Receive Connector via PowerShell

Open an elevated Exchange Management Shell on your Mailbox server. Never edit default system connectors; create a dedicated connector bound to the Frontend Transport service:
POWERSHELL
# Parameters for Dedicated Scoped Relay Connector
$ServerName = "EX01"
$ConnectorName = "Internal-App-Relay-Scoped"
$AllowedIPs = @("192.168.10.50", "192.168.10.51", "10.0.5.20-10.0.5.25")

New-ReceiveConnector -Name $ConnectorName `
                     -Server $ServerName `
                     -Type Custom `
                     -TransportRole FrontendTransport `
                     -Usage Custom `
                     -Bindings "0.0.0.0:25" `
                     -RemoteIPRanges $AllowedIPs `
                     -AuthMechanism None `
                     -PermissionGroups AnonymousUsers `
                     -ExtendedProtectionTokenChecking None

Write-Host "[+] Receive Connector '$ConnectorName' created on $ServerName." -ForegroundColor Green

2. Grant External Relay Rights to Anonymous Users on that Connector

By default, AnonymousUsers can only submit email to recipients inside your authoritative accepted domains. To permit sending to external recipients (e.g., dispatching alert emails to a vendor or on-call staff):
POWERSHELL
# Grant the 'Accept Any Recipient' permission to the Anonymous Logon security principal
Get-ReceiveConnector "$ServerName\$ConnectorName" | `
    Add-ADPermission -User "NT AUTHORITY\ANONYMOUS LOGON" `
                     -ExtendedRights "ms-Exch-SMTP-Accept-Any-Recipient"

Write-Host "[✔] External relay rights granted strictly for IPs in: $($AllowedIPs -join ', ')" -ForegroundColor Green

Testing SMTP Relay via Modern PowerShell (Replacing Telnet)

Legacy telnet commands are clumsy and do not support TLS handshakes. Use the following robust PowerShell script to test relay functionality:

POWERSHELL
# Script: Test-SmtpRelay.ps1
# Description: Validates mail relay against an internal Exchange connector

$SmtpServer = "192.168.10.10" # Internal Exchange IP or DNS
$From       = "[email protected]"
$To         = "[email protected]"
$Subject    = "SMTP Relay Smoke Test - $(Get-Date -Format 'yyyy-MM-dd HH:mm')"
$Body       = "This is an automated smoke test validating scoped anonymous SMTP relay."

$SmtpClient = New-Object System.Net.Mail.SmtpClient($SmtpServer, 25)
$SmtpClient.EnableSsl = $false # Internal plain or opportunistic TLS
$SmtpClient.Timeout   = 10000

$MailMessage = New-Object System.Net.Mail.MailMessage($From, $To, $Subject, $Body)

try {
    Write-Host "Sending test message through $SmtpServer..." -ForegroundColor Cyan
    $SmtpClient.Send($MailMessage)
    Write-Host "[✔] Message dispatched successfully! Relay verified." -ForegroundColor Green
}
catch {
    Write-Error "SMTP Relay failed: $_"
}
finally {
    $MailMessage.Dispose()
    $SmtpClient.Dispose()
}

Modern Alternative: Exchange Online High Volume Email (HVE)

If you have migrated completely to Microsoft 365, avoid running an on-premises Exchange server just for SMTP relay. Use Exchange Online High Volume Email (HVE):

  1. Navigate to Exchange Admin Center (EAC): Go to Mail flow > High Volume Email (HVE).
  2. Create an HVE User: Register an account (e.g., [email protected]).
  3. Configure Appliances:

- SMTP Host: smtp-hve.office365.com - Port: 587 - TLS: STARTTLS (TLS 1.2 required) - Authentication: Basic SMTP AUTH credentials generated specifically for the HVE account. - Capacity: Supports up to 100,000 internal recipients daily with detailed consumption dashboards.


Security Audit Checklist

ItemBest PracticeRisk If Ignored
IP WhitelistSingle IPs or small /29 CIDR blocksSubnet compromise turns server into Open Relay
Sender FilteringMatch From: header with accepted domainsMalicious devices can spoof CEO/executive addresses
Connector SeparationKeep dedicated connector on Port 25; do not modify Default FrontendDisrupts normal inbound MX mail delivery
SPF AlignmentInclude the server's public egress IP in v=spf1External recipients reject or junk relayed messages