Lessons learned from an Internet outage

We currently co-locate a small number of servers for off site backup as well as anti-spam appliances; this is part of our Think Local initiative.

Just after ceremony of our daughter’s graduation from culinary school, I was paged with the message the Internet connectivity was down to the facility.

I was able to verify the loss of Internet connectivity as we drove home; and then it became the dance between getting updates from the co-location facility in Lancaster, PA and providing updates to our customers.

This was the very first major outage the co-location facility has had in years; and, the very first outage that we experience since becoming their customer a little over two years ago.

The outage started shortly after 5:00 PM Eastern Time, and as it started to head past 11:00 PM, we had were faced with the following issues:

1. Off site backups would fail (as the backup server in Lancaster, PA could not reach out to the servers for which backup is scheduled).
2. Email to our customers that go through the anti-spam appliance would bounce.

Thank Jesus, we did have a Plan B for the mail appliance.

We would redirect the MX (mail exchange) record to point directly to our mail servers rather than the anti-spam appliance. While there would be an increase in spam delivered, at least mail delivery would have minimal impact.

While we did wait as long as possible to see if we had to implement plan b, we ended up doing so shortly before 3 AM Eastern Time.

Thankfully, connectivity with the local co-location facility was restored around noon time the next day; and we were able to shift gears back to the anti-spam appliance… and we only lost one day of not backing up.

While we had Plan B planned out, we got to experience the hick ups (oh, we forgot about customer abc that routes to a different mail server); and we updated our documentation if Plan B is ever needed again.

Please share yours in the comment section below.