I’ve discovered that there are always lots of random things that can wreck a good weekend… For example, getting alarms, and emails at 0700 on a Sunday morning reporting possible power issues, then more alarms at 0900 confirming a power outage.
After talking with the building supervisor, and finding out that a fire alarm had triggered the power outage, I raced into work, and decided to take a look, and see what we could get back up, and how soon1. When arriving, I find the doors won’t unlock2, the building generator running, and other odd things, like flickering lights.
After gaining entrance into the building, I start to look around, and from the looks of it, we have power, but the server room isn’t coming back up. Doing some tests on the UPS, shows the UPS is okay, but won’t power up. Server room lights wouldn’t power up either, this is getting concerning.
So we tweaked with the UPS a bit, checked the breakers, called the building supervisor again, who finally3 decided to come in and check things out. By this time we decided to call the UPS manufacturer (Eaton), and get their support on the line to find out if we had something wrong with the UPS. This is where we find out our support contract wasn’t renewed, and it’s going to cost $288/hour, with a minimum of 4 hour charge if they sent somebody out… fun.
The building supervisor finally arrives as we’re doing diagnostics on the UPS, and is a little puzzled as to why the generator is still running. After a few more hours of playing around, finally calls TXU, they send out a repair team to take a look. We figured out the issue wasn’t UPS related when we couldn’t get half the printers in the office to come back online, and the lights were flickering in some offices.
Doing additional testing showed we were running at half the voltage we should have been on one of the legs coming into the building… That’s not a good sign… So TXU started to take a look, and it became clear pretty quickly what the cause of the problem was.
Apparently the little slithering thing had crawled up into the buildings transformer, and shorted the circuit, cooking himself in the process… I ended up getting home at about 1630… So the smallest of things can wreck a good weekend.