Did you now that cPanel’s default configuration of the Exim mail server can retry a failed email message nearly 150 times?
Yes, that’s right 150 times.
If Yahoo! delays an email just once, your system can spiral out of control with nearly 150 delivery attempts for the email. Just 100 failed emails can quickly balloon into 15,000 delivery attempts over 4 days. This high rate of deliveries can trigger Yahoo!’s email blacklist filters, causing you further email problems. (If you have already been blacklisted, check out our How to Remove Your IP from Yahoo!’s Blacklist post).
For example, here we see Yahoo! blocking delivery with a 421 temporarily deferred error. Due to a spam incident, this server tried to deliver email to Yahoo! 450,000 times in 48 hours, triggering Yahoo’s blacklist. If this has happened to you, learn how to remove your IP from Yahoo’s Blacklist.
Fortunately, we can fix this problem with Exim’s hosts_max_try_hardlimit.
Yahoo! Mail and 421 SMTP Errors
When Yahoo! suspects your email is spam, their servers send a 421 SMTP error. The 421 code means, “try again later.” Yahoo’s help page states:
If you regularly receive 421 SMTP errors when sending email to Yahoo Mail addresses, check your server logs for unusual traffic patterns or objectionable message content.
The initial trigger for the 421 error is usually spamming, forwarding spam or someone sending a gigantic newsletter through your server, and this is when the Exim configuration becomes an issue.
Exim’s Retry Methods
When Exim received a 421 error, the mail server re-queues the message for later delivery according to the retry configuration:
* * F,2h,15m; G,16h,1h,1.5; F,4d,8h
What this means is that if there is a failure, retry the message every 15 minutes for 2 hours (8 attempts). Then retry at geometric intervals (G) for 16 hours. I won’t bore you with the math, but that roughly translates to another 7 delivery attempts. Finally, if the message is still not delivered after this 16 hours, the message sits in the queue for 4 days with deliveries about every 8 hours ( ~ 13 attempts).
In total, this is 28 delivery attempts.
Here’s a snapshot of the retry block in cPanel’s advanced Exim configuration tool.
So for a single 421 error, Exim retries 28 times, but how does this explode to nearly 150?
Multihomed Servers
Yahoo! and many other email providers have multiple MX records, each with multiple IP addresses.
dig +short yahoo.com mx 1 mta7.am0.yahoodns.net. 1 mta6.am0.yahoodns.net. 1 mta5.am0.yahoodns.net.
There are 3 MX records. However, each MX record has multiple IPs:
dig +short mta7.am0.yahoodns.net 66.218.85.52 67.195.204.72 67.195.228.106 67.195.228.109 67.195.228.110 74.6.137.65 98.136.96.76 98.136.96.91
When Exim looks up these IP addresses, the mail server places every IP on a host list. Exim essentially treats each IP as a host. If one host fails, the system tries another host. By default, Exim tries 5 hosts per delivery attempt as set in the host_max_try variable.
You can see this in your logs:
2019-08-18 07:11:55 2izMuV-00057P-5e SMTP connection outbound 1536137515 2izMuV-00057P-5e localdomain.com [email protected] 2019-08-18 07:11:55 2izMuV-00057P-5e H=mta7.am0.yahoodns.net [67.195.228.109]: SMTP error from remote mail server after pipelined MAIL FROM:<[email protected]> SIZE=12782: 421 4.7.0 [TSS04] Messages from 198.15.70.42 temporarily deferred due to user complaints - 4.16.55.1; see https://help.yahoo.com/kb/postmaster/SLN3434.html 2019-08-18 07:11:56 2izMuV-00057P-5e H=mta7.am0.yahoodns.net [67.195.228.110]: SMTP error from remote mail server after pipelined MAIL FROM:<[email protected]> SIZE=12782: 421 4.7.0 [TSS04] Messages from 198.15.70.42 temporarily deferred due to user complaints - 4.16.55.1; see https://help.yahoo.com/kb/postmaster/SLN3434.html 2019-08-18 07:11:56 2izMuV-00057P-5e H=mta7.am0.yahoodns.net [67.195.228.106]: SMTP error from remote mail server after pipelined MAIL FROM:<[email protected]> SIZE=12782: 421 4.7.0 [TSS04] Messages from 198.15.70.42 temporarily deferred due to user complaints - 4.16.55.1; see https://help.yahoo.com/kb/postmaster/SLN3434.html 2019-08-18 07:11:56 2izMuV-00057P-5e H=mta7.am0.yahoodns.net [67.195.228.111]: SMTP error from remote mail server after pipelined MAIL FROM:<[email protected]> SIZE=12782: 421 4.7.0 [TSS04] Messages from 198.15.70.42 temporarily deferred due to user complaints - 4.16.55.1; see https://help.yahoo.com/kb/postmaster/SLN3434.html 2019-08-18 07:11:57 2izMuV-00057P-5e H=mta7.am0.yahoodns.net [67.195.228.94]: SMTP error from remote mail server after pipelined MAIL FROM:<[email protected]> SIZE=12782: 421 4.7.0 [TSS04] Messages from 198.15.70.42 temporarily deferred due to user complaints - 4.16.55.1; see https://help.yahoo.com/kb/postmaster/SLN3434.html
This pattern continues for 4 days. Due to the internal rescheduling mechanism, you may even see Exim attempt 10-20 deliveries within a short period.
On Yahoo’s end, they are tracking all of the failed delivery attempts. Your server keeps trying over and over — nearly 150-200 times in 4 days. This behavior is very similar to a spambot. As a result, Yahoo continues to block your IP address.
The Fix
Fortunately, there is a fix for this issue by limiting the number of IPs that Exim will try.
One solution is to update the default retry block rules to fail messages more quickly. (See our tutorial on how to change Exim’s retry rules for more detail. )
The second change, which is designed specifically for this issue, is to set the hosts_max_try_hardlimit
The hosts_max_try_hardlimit option was added to help with this problem. Exim never tries more than this number of IP addresses; if it hits this limit and they are all timed out, the email address is bounced, even though not all possible IP addresses have been tried.
This setting resolves the multihomed host problem by setting a strict limit. If you have delivery issues with Yahoo!, I recommend you set hosts_max_try_hardlimit to 1 if you are having issues. Otherwise, you can leave the default.
On cPanel, you will need to manually edit the exim.conf file to include this line. You place it under the smtp remote transports.
remote_smtp:
driver = smtp
interface = <; ${if > {${extract{size}{${stat:/etc/mailips}}}}{0}{${lookup{${lc:${perl{get_message_sender_domain}}}}lsearch{/etc/mailips}{$value}{${lookup{${lc:$original_domain}}lsearch{/etc/mailips}{$value}{${lookup{${perl{get_sender_from_uid}}}lsearch*{/etc/mailips}{$value}{}}}}}}}}
helo_data = ${if > {${extract{size}{${stat:/etc/mailhelo}}}}{0}{${lookup{${lc:${perl{get_message_sender_domain}}}}lsearch{/etc/mailhelo}{$value}{${lookup{${lc:$original_domain}}lsearch{/etc/mailhelo}{$value}{${lookup{${perl{get_sender_from_uid}}}lsearch*{/etc/mailhelo}{$value}{$primary_hostname}}}}}}}{$primary_hostname}}
hosts_try_chunking = 198.51.100.1
hosts_max_try_hardlimit = 1
If you are using dkim, you will also want to place this under the dkim_remote_smtp transport sections as well. Once you have added the line, restart exim.
1980’s Settings & Today’s Networks
The problem with email servers is that many defaults are the same as they were in the 1980’s. Then, a company’s email server may not be connected 24/7. Servers would connect periodically to the internet and fetch the mail, so long queue times make sense. Today, if an email server cannot send an email within a few minutes, the message cannot be delivered. There’s no sense in retrying 150-200 times as it makes your server look like spam source.
Summary
In practice, I don’t make changes to the host max retry settings unless you have delivery issues to Yahoo!. I do update the general RETRYBLOCK to fail deliveries faster. Note that unlike real-time email blacklists, Yahoo’s Blacklist does not publish a lookup tool. You know you are blacklisted when you get an email bounce from Yahoo.