We’re back up and running!

As you’ve undoubtedly noticed, OSAlert has been down most of the day. We’re back up now, running like we always do. We rarely have this kind of extensive downtime, so in the interest of full disclosure, here’s what happened. If you’re expecting some sexy story – think again.

After I notified our sysadmin that OSAlert was dishing out 324s, he got back to me they were already working on it. We had the same issue for a really short while a few days ago, but it got sorted then. Our sysadmin, Jon Jensen, over at Endpoint checked remotely what the heck was going on, and confirmed it wasn’t actually a software issue – it was a hardware issue, more specifically, a RAID issue. Had the RAID controller failed?

Since this obviously couldn’t be addressed remotely, Jon contacted the datacentre, and when they got back to him, the true extent of the problem was revealed. The quote from the datacenter tech? “We have fixed the cabling issue with your server and your server is now back online.”

Believe it or not, the connections of the disk cables to the controller had come loose. After the first outage happened a few days ago, they were apparently still somewhat connected, only to fully come loose today. Quite a feat, for an immobile server in an immobile rack.

The funny coincidence is that this all happened while we were also working to set up a playground subdomain for the next version of OSAlert we’re currently working on. Maybe it’s a sign…

34 Comments

  1. 2012-02-24 7:16 pm
    • 2012-02-25 10:40 am
  2. 2012-02-24 7:17 pm
    • 2012-02-24 7:29 pm
  3. 2012-02-24 7:29 pm
    • 2012-02-25 10:42 am
      • 2012-02-25 12:32 pm
  4. 2012-02-24 7:31 pm
    • 2012-02-25 2:51 am
    • 2012-02-25 11:00 am
      • 2012-02-25 9:12 pm
  5. 2012-02-24 7:35 pm
    • 2012-02-24 7:45 pm
      • 2012-02-24 8:00 pm
        • 2012-02-24 8:33 pm
          • 2012-02-24 8:43 pm
          • 2012-02-24 11:19 pm
    • 2012-02-24 9:40 pm
  6. 2012-02-24 7:57 pm
    • 2012-02-24 8:54 pm
      • 2012-02-24 9:44 pm
      • 2012-02-25 3:14 am
  7. 2012-02-24 8:54 pm
    • 2012-02-24 9:08 pm
      • 2012-02-24 9:50 pm
        • 2012-02-24 11:24 pm
          • 2012-02-25 7:25 am
  8. 2012-02-24 8:57 pm
    • 2012-02-25 12:45 pm
  9. 2012-02-24 11:15 pm
    • 2012-02-24 11:53 pm
  10. 2012-02-26 4:03 am
    • 2012-02-27 8:38 am