Well, I am apparently unable to take a vacation… ever.
As I previously mentioned, Thursday last week, at about 0200 our main line went down. It was down for about an hour until I got it back up again. Later that morning, I spoke with our line provider SpeakEasy, they did some line tests, and reported a lot of noise on the line. They decided to engage our line provider to have a look at the line. They got back to us and reported the line would have to be replaced, and have set about doing it. This was where it was left on Thursday. The line had stabilized, all was well…
Tuesday afternoon, while on vacation back home, my boss called reporting total line loss, and no working connectivity. This was causing so much trouble that we were about to break SLAs with our customers, and we’d have to start paying them back money. This meant it would be a long day. I had to figure a way to move all existing connectivity to our new fiber line, without interrupting service.
This ended up being easier than I had ever hoped. A simple IP address change on the existing firewall (which is linux) to a spare IP on the new line. I then called our technical account rep at AT&T, and asked for an emergency change request on the firewall to allow the new IP address to establish a VPN tunnel to our hosting environment. This was done in about 20 minutes. within 45 minutes, I had the server moved, the vpn tunnel reestablished, and a faster, more happier network in place.
I then spent the rest of the night fixing other minor IP address changes, configuring new monitoring policies, and clearing everything is okay with our data and development teams to make sure everything is running smoothly. Then, about 2 hours ago, I received a call from SpeakEasy reporting that our line provider has completed repairs on the line (only 1 week later), and that I can test. This is of course a little difficult now because I am at “home”. oh well… breaking SLAs for 3 complete days, 1500 customers… I think I’d have been jobless.