I have noticed over the last few days that we have had Roadrunner messages in our main e-mail gateway stuck in our queue with the following error:
56FE71501A1C 1695 Fri Jul 30 08:32:19 noreply@nowhere.com
(host hrndva-smtpin02.mail.rr.com[71.74.56.244] refused to talk to me: 421 4.7.1 - Connection refused - - Too many concurrent connections from source IP)
Due to Roadrunner’s rather draconian inbound rate limit policy, we receive this bounce error almost constantly. A fairly large number of the e-mail that is sent out over our servers is headed for Roadrunner’s system. Given that much of the traffic is generated from local church mailing lists, a delay of a few hours usually isn’t that big of a deal. At least, it hasn’t been up until this week. Once I got in this morning however, I saw that we had almost 5,000 messages in the queue with the same error. Alarm bells went off and the IT Office shifted into DEFCON 1.
Searching through the 5,000 messages quickly showed me that there was one sender who was doing most of the work. Grepping through /var/log/maillog quickly showed me which server the traffic was coming from. Once I had this information handy I shut down the main gateway server to stop the bleeding until I could figure out what was going on. I headed over to the server in question and started searching through those logs as well.
Turns out that there was an old Squirrelmail plugin that the spammer was using to generate plain text spam messages internally. Specifically these were nigerian phishing scheme messages. Since they are plain text, coming to and from legitimate addresses and were constantly changing, they were very difficult for our spam filters to stop. Once I updated the plugins, cleaned out all of the mail queues in question and restarted the affected services we were back in action.
The only problem remaining was to convince my fellow e-mail administrators that we were no longer sending spam. All of the major ISP’s use different filtering systems, real time blacklists (RBL’s) and their own internal policies. This meant visiting each ISP listed in the queue as a bounced message and reporting to them individually that the issue was resolved.
Along the way I noticed that there were two agencies that many of the ISP’s use as a clearing house for online e-mail system reputation. You can sign up your organization and they will independently verify that you are not a spammer. Once you are on their list, the ISP’s that subscribe to their services can verify that you are a legitmate sender. It amounts to a shake down more or less. You have to pay to play. Given the importance of our e-mail systems I decided to go ahead and sign us up. If you need to do the same with your mail servers then visit these companies:
- E-mailReg.org – This service charges $20 for one year of service. After verifying domain ownership they say it will take several days.
- ReturnPath.net – This is the big service that a lot of the major ISP’s use. They charge for their service based on how big your organization is. We received the non-profit discount and paid $200 for the application fee with no monthly fee. This system will take several weeks to verify. You really should sign up for this service when times are good.
All is well now. The spam flow has been stopped and all of our queues are cleaned out. All we can do now is wait on Roadrunner’s rate limits to time out and allow us to resume sending messages.
