There’s nothing better than a dose of high octane stress to kick off a week just right.  I came in this morning to find out that no one could connect to the server anymore.  Our office was flooded with calls wondering what was wrong with the e-mail server.  In the past people could have clicked past the expiration error and kept on trucking.  I quickly renewed the certificate, downloaded and installed it.  Then the real problem started.  All of our clients could now access the server but they were getting trust errors.  Turns out the new Thawte certificates need to have an intermediate CA certificate installed.

Thawte uses Intermediate CAs to enhance the security of SSL and Code Signing certificates. Installing the correct Intermediate CAs or CA bundle for the certificate being used is absolutely essential to ensure that users don’t see certificate errors when visiting a website or running software secured with a Thawte certificate.

I didn’t know about this since it had changed in the last year.  After running it by their technical support (they give great chat by the way) I was pointed to an article discussing the issue.  Turns out this new requirement was implemented on June 27th, 2010.  I downloaded the required certificate and added the following line to ssl.conf:

SSLCACertificateFile /usr/local/ssl/crt/cabundle.crt

One quick Apache restart and all is well.  Now it’s noon on Monday.  Time to get the week started!

Safe Searching!

Google has released the beta version of their https encrypted search website.  Google probably didn’t encrypt their searches from the beginning due to the increased overhead of the https protocol.  After several test searches I cannot tell a difference between the http equivalent.  Now we can all search whatever we want in the coffee shop without much worry of wireless sniffers watching our every move.  Very impressive!

GNU Mailman

We have a long running history with this software.  Sometime around 2003 I was tasked with setting up a mailing list solution for the Conference.  Several of our local churches had also expressed interest so I had to find something cheap, scalable and fast.  GNU Mailman was the perfect choice.  It’s free and open source software, you can continue throwing lists at it and it supports lists of all sizes.

The list server is my oldest Linux installation.  I was a lowly Windows admin at the time so my good friend Alan Swartz helped me with the original Red Hat installation.  My how times have changed.  Back then I had a brief list of commands to create new lists, reboot the server and perform a few basic administrative tasks.  I wish I had kept a copy of that original handwritten list but alas, it is lost to the sands of time.  This software has proven robust over the years as it has moved across several physical computers and 3-4 different Linux distributions.

Victims of our own success.

It would seem that too much of a good thing always lead to problems.  The mailing list server maintains an archive of all of the e-mail that is sent over each of the mailing lists.  These files grow over time as new messages are sent.  Over time disk space can become a problem.  It took us several years of constant usage to amass a corpus of around 80 gigabytes (GB).  Mailman must have changed how it stores e-mail because over the course of a year or so we shot up to around 280 GB.  Maybe people realized that you can send attachments to the lists?  Once things get back to normal I plan to dig into why these list archives are growing so quickly.

Everyone loves a good history lesson but why is the server down today?  The simple answer is that the hard drive is full (again).  Once it fills up the mailman daemon stops responding.  Since I am out of the office it could take me a good while to discover that the system is down.  That’s why I decided to go ahead and replace it.

On June 17th the system went down due to a full hard drive.  With the building move coming up soon I decided to try temporarily remove the larger archives from the internal mailing lists.  This would free up enough hard disk space to keep the server running (hopefully) until well after the building move when I could properly schedule an outage.  I’ve been monitoring the disk utilization since then, moving archives as I can.  Unfortunately, I’ve moved all of the larger ones and was forced to move forward with plans to switch the drive.

Last night I pulled the 320 GB drive and replaced it with a 1.5 Terabyte (TB) drive.  It takes awhile to copy the archives back to the new drive however.  Overnight 60 GB of the 280 GB data store copied.  I expect progress all day and will bring the system back online as soon as I can.  Hopefully this will buy us a good bit of time before I have to permanently retire some of the archives.

Update: Friday, July 9th 2010 @ 12:17 PM
The list server is back up!  We have plenty of available disk space now.  I’m hoping that this one will last us a good while.  I still need to research what is eating so much disk space but moving forward we should be in good shape!

We launched Google.cn in January 2006 in the belief that the benefits of increased access to information for people in China and a more open Internet outweighed our discomfort in agreeing to censor some results. At the time we made clear that “we will carefully monitor conditions in China, including new laws and other restrictions on our services. If we determine that we are unable to achieve the objectives outlined we will not hesitate to reconsider our approach to China.”

These attacks and the surveillance they have uncovered–combined with the attempts over the past year to further limit free speech on the web–have led us to conclude that we should review the feasibility of our business operations in China. We have decided we are no longer willing to continue censoring our results on Google.cn, and so over the next few weeks we will be discussing with the Chinese government the basis on which we could operate an unfiltered search engine within the law, if at all. We recognize that this may well mean having to shut down Google.cn, and potentially our offices in China.

Subject: Introspection Online
Date: Mon, 8 Sep 1997 11:12:25 -0400 (EDT)

Introspection Online

http://www.angelfire.com/nc/mywebpage1/lobby.html

(Link inactive 27 May 2004)

Introspection Online is a free email-based newletter published
independently by a collective of devoted individuals and dedicated to
literary writing and opinion.

Individuals can subscribe to Introspection Online by sending an email
request to:  Writermail@aol.com 

Contact:

Douglas Ward, Editor, Mathiel@aol.com

This is the week that Geocities went dark forever.  There is no backup or spare copy.  It’s simply gone.  With it goes a big portion of the history of the internet.  That is a true shame.  Lucky for us some people saved at least some of it for historical purposes.

When Yahoo! switched off the servers for GeoCities, the Web posting service, on Oct. 27, some 7 million of the Internet’s first websites went dark forever. The bulk of these were people’s personal home pages, which were pulled offline with no backup and no permanent record of those users’ frenetic early forays online.

Now a ragtag effort by several groups of budding computer historians is feverishly — and angrily — trying to bring as much as they can back online.

…

ArchiveTeam is still sorting through the data, but Scott estimates that he was able to save more than a million accounts, which translates to more than 2 terabytes of data (about 20 average computer hard drives). And he wasn’t alone — Scott says that four or five others were working to save GeoCities. One of these people, Jacques Mattheij, managed to get nearly 2 million accounts, operating just eight machines out of the Netherlands.

I’m surprised there wasn’t some deal struck with Archive.org or some other institution.  At least someone is doing something.  Kudos to the Archive Team!