56FE71501A1C 1695 Fri Jul 30 08:32:19 noreply@nowhere.com
(host hrndva-smtpin02.mail.rr.com[71.74.56.244] refused to talk to me: 421 4.7.1 - Connection refused - - Too many concurrent connections from source IP)

Due to Roadrunner’s rather draconian inbound rate limit policy, we receive this bounce error almost constantly. A fairly large number of the e-mail that is sent out over our servers is headed for Roadrunner’s system. Given that much of the traffic is generated from local church mailing lists, a delay of a few hours usually isn’t that big of a deal. At least, it hasn’t been up until this week. Once I got in this morning however, I saw that we had almost 5,000 messages in the queue with the same error. Alarm bells went off and the IT Office shifted into DEFCON 1.

Searching through the 5,000 messages quickly showed me that there was one sender who was doing most of the work. Grepping through /var/log/maillog quickly showed me which server the traffic was coming from. Once I had this information handy I shut down the main gateway server to stop the bleeding until I could figure out what was going on. I headed over to the server in question and started searching through those logs as well.

Turns out that there was an old Squirrelmail plugin that the spammer was using to generate plain text spam messages internally. Specifically these were nigerian phishing scheme messages. Since they are plain text, coming to and from legitimate addresses and were constantly changing, they were very difficult for our spam filters to stop. Once I updated the plugins, cleaned out all of the mail queues in question and restarted the affected services we were back in action.

The only problem remaining was to convince my fellow e-mail administrators that we were no longer sending spam. All of the major ISP’s use different filtering systems, real time blacklists (RBL’s) and their own internal policies. This meant visiting each ISP listed in the queue as a bounced message and reporting to them individually that the issue was resolved.

Along the way I noticed that there were two agencies that many of the ISP’s use as a clearing house for online e-mail system reputation. You can sign up your organization and they will independently verify that you are not a spammer. Once you are on their list, the ISP’s that subscribe to their services can verify that you are a legitmate sender. It amounts to a shake down more or less. You have to pay to play. Given the importance of our e-mail systems I decided to go ahead and sign us up. If you need to do the same with your mail servers then visit these companies:

• E-mailReg.org – This service charges $20 for one year of service. After verifying domain ownership they say it will take several days.
• ReturnPath.net – This is the big service that a lot of the major ISP’s use. They charge for their service based on how big your organization is. We received the non-profit discount and paid $200 for the application fee with no monthly fee. This system will take several weeks to verify. You really should sign up for this service when times are good.

All is well now. The spam flow has been stopped and all of our queues are cleaned out. All we can do now is wait on Roadrunner’s rate limits to time out and allow us to resume sending messages.

Safe Searching!

Google has released the beta version of their https encrypted search website.  Google probably didn’t encrypt their searches from the beginning due to the increased overhead of the https protocol.  After several test searches I cannot tell a difference between the http equivalent.  Now we can all search whatever we want in the coffee shop without much worry of wireless sniffers watching our every move.  Very impressive!

GNU Mailman

We have a long running history with this software.  Sometime around 2003 I was tasked with setting up a mailing list solution for the Conference.  Several of our local churches had also expressed interest so I had to find something cheap, scalable and fast.  GNU Mailman was the perfect choice.  It’s free and open source software, you can continue throwing lists at it and it supports lists of all sizes.

The list server is my oldest Linux installation.  I was a lowly Windows admin at the time so my good friend Alan Swartz helped me with the original Red Hat installation.  My how times have changed.  Back then I had a brief list of commands to create new lists, reboot the server and perform a few basic administrative tasks.  I wish I had kept a copy of that original handwritten list but alas, it is lost to the sands of time.  This software has proven robust over the years as it has moved across several physical computers and 3-4 different Linux distributions.

Victims of our own success.

It would seem that too much of a good thing always lead to problems.  The mailing list server maintains an archive of all of the e-mail that is sent over each of the mailing lists.  These files grow over time as new messages are sent.  Over time disk space can become a problem.  It took us several years of constant usage to amass a corpus of around 80 gigabytes (GB).  Mailman must have changed how it stores e-mail because over the course of a year or so we shot up to around 280 GB.  Maybe people realized that you can send attachments to the lists?  Once things get back to normal I plan to dig into why these list archives are growing so quickly.

Everyone loves a good history lesson but why is the server down today?  The simple answer is that the hard drive is full (again).  Once it fills up the mailman daemon stops responding.  Since I am out of the office it could take me a good while to discover that the system is down.  That’s why I decided to go ahead and replace it.

On June 17th the system went down due to a full hard drive.  With the building move coming up soon I decided to try temporarily remove the larger archives from the internal mailing lists.  This would free up enough hard disk space to keep the server running (hopefully) until well after the building move when I could properly schedule an outage.  I’ve been monitoring the disk utilization since then, moving archives as I can.  Unfortunately, I’ve moved all of the larger ones and was forced to move forward with plans to switch the drive.

Last night I pulled the 320 GB drive and replaced it with a 1.5 Terabyte (TB) drive.  It takes awhile to copy the archives back to the new drive however.  Overnight 60 GB of the 280 GB data store copied.  I expect progress all day and will bring the system back online as soon as I can.  Hopefully this will buy us a good bit of time before I have to permanently retire some of the archives.

Update: Friday, July 9th 2010 @ 12:17 PM
The list server is back up!  We have plenty of available disk space now.  I’m hoping that this one will last us a good while.  I still need to research what is eating so much disk space but moving forward we should be in good shape!

Scotty, we need more power!

I particularly enjoyed working with my co-workers in the News Room this year.  Since we had the extra day the entire operation had a significantly lower stress level throughout for everyone.  Some things broke and had to be fixed.  Some things had us running in circles for awhile.  We had our challenges like we do every year but this year has to stand out as one of the better ones.  We had time to actually talk to each other during the week!

Derek and I get stoked at the Media Center booth.

One of the big new items to come out of this year’s work is that we will be meeting at the Raleigh Convention Center for the next two years!  I am excited about being able to sleep in my own bed for part of the week but I will miss Greenville.  From an operations standpoint the Greenville Convention Center fit our needs quite well.  It remains to be seen how things will work in Raleigh but I’m not worrying about that this year.

Bishop Gwinn’s State of The Church Address stands out as one of the highlights of conference in my mind.  He delivered a powerful message to the people about a variety of topics including cross racial appointments.  I have never seen him get so emotional before.

State of the Church Address – Bishop Gwinn from NCCUMC.org on Vimeo.

Another of my highlights has to be the African Children’s Choir.  What a joy they were!  This group of children from war torn Africa inspired and moved everyone in the crowd.  How can anyone watch this and now smile?  I shot this video from backstage with my Droid.  Apologies for the poor quality…

That brings us to the end of the 2010 North Carolina Annual Conference.  I look forward to seeing everyone next year.  Now it’s time to break all of this stuff down and head home.  I’m going to take a day of rest tomorrow and then dive right into the building move.  June is a busy month!

Affected browsers: Google Chrome, Mozilla Firefox, Internet Explorer and Safari.  All are running the latest patches as of this writing.

Steps to remediate for Windows XP users:

1. Start – Run
2. Run the program “cmd” for the command line
3. Enter “ipconfig /flushdns” and hit Enter
4. Restart the browser

Steps to remediate for Windows Vista/7 users:

1. Start – All Programs – Accessories
2. Right click on command prompt and select Run As Administrator
3. Enter “ipconfig /flushdns” and hit Enter
4. Restart the browser

I have seen some small issues with Ubuntu and Mac laptops.  We resolved those by dumping the browser cache, restarting the network connections and restarting the browser.  I will post an update as I learn more.

Update: June 10th @ 5:24 PM

I’ve done a good bit of googling and found out that the issue is most likely linked to our brand spanking new Linksys WRT320N wireless router (relevant threads can be found here and here).  Apparently that entire family of routers has trouble with DNS requests.  I didn’t see a sticker on the box when I bought it that said something along the lines of “I suck at DNS.”  Who knew?  I updated the firmware at our dinner break.  We’ll see how it goes from here.

Multiple routers providing internet access.

We run a pretty wide range of hardware in the News Room.  You can see anything from Linux netbooks to Mac and Windows laptops.  We also support an e-mail station of several Ubuntu 10.04 workstations. Throw in about 15-20 employees, several wireless networks, 2,000 attendees and one large facility and you have a complex project!

George Speake and Derek Leek work together on a project.

This year the set-up time took about as long as I expected it would.  The nice thing we have this year that we have not had in previous years is an extra day to prepare.  Now that I have most of the major work finished tomorrow should be a fairly light day.  I’m hoping to catch up on some e-mail and continue coordinating vendors for the building move in two weeks.  We’ll see how it goes though.  As of right now all systems are green!

Gmail interface - Work e-mail address!

Who could say no to an offer like that?  We are looking at the collaborative possibilities of having over 2,000 active accounts in one Google Apps system.  Our current e-mail infrastructure is a mixture of Microsoft and open source products.  We started off with a small Windows e-mail server and have grown over the years to support over 2,000 mailboxes.  We decided to go with a few different solutions for different segments of this user base over time as needed.  In the end we have five servers all working together to run and manage our communications infrastructure.  It’s a mess.  It’s also rather burdensome to administer.  Enter Google Apps with its unified management system, familiar end user facing interface, compelling and easy to use overall setup and the choice to switch is fairly easy.  We started off with about 10 beta users and then migrated the entire Communications and Information Technology offices.  Once we were convinced that the system was working well we began the roll out process to internal staff.  We are up to about 150 active accounts now but we have a long way to go.

What are some of the major benefits of switching to this system?

• We reduce our support surface from five different servers, operating systems and mailbox formats down to one integrated interface.
• We will have access to an whole host of collaborative tools that was not previously possible.
• Our end users will have reduced confusion.  Many times they call us and have forgotten which server they are hosted on (so do we!).
• We don’t have to write documentation anymore.  We just link from Google’s support website!
• We no longer have to constantly maintain our e-mail gateway anti-virus and anti-spam scanners.
• Microsoft Outlook integrates into the Google Apps system.  We have a good number of users that wish to continue using Microsoft Outlook instead of the web interface.

Google Apps allows a custom logo for each organization.

What kind of things can we do with this new system?  Here’s a few ideas:

• Easily share user calendars so that co-workers may know when they are available for meetings.
• Set up shared documents, spreadsheets and presentations for collaborative work within offices.  These documents can also be shared with the entire Conference or with all visitors to our website.
• Set up Google Sites for project based collaboration (rather than just individual documents).
• Easily manage e-mail and make use of the labeling system for filing away old messages for later retrieval.
• Easily set up a system group that I can use to communicate with all of my end users simultaneously.

What other ways can we use this?  We are concentrating on the migration from our existing systems now and will move on to feature implementation later.  I’m curious to hear other ideas as to what we can actually do with this system once we have everything in place.  I’ll post updates as we move along in the implementation process.  We are very excited about all that we can do with this new system!