I am, once again, sitting on a 0200 conference call for an emergency change request to push a bug fix to production. I’m bubbling with frustration right now because once again, we spend 40 minutes unsure of:
- How to test the bug
- If the bug was what it was
- Who should be testing
- What the successful result should have been
The problem with tonights update was pretty simple. A small stored procedure change, executed in a few seconds, by the corporate DBAs. The first few minutes of testing was filled with… “well I ran it, but I’m not sure, so get the other guy” statements. When he jumped on, he did the same thing, announced to the world it worked, but didn’t believe the code change had been run before the first guy executed the code. This of course was not true, but he was pretty adamant about this.
Then the first guy proceeds to ask if the VPN would impact it, to which the second guy said it might do, and you should VPN into the corporate network. I proceed to explain that access to the production network from the corporate office would be identical, with the exception of source IP (I didn’t explain that bit as I feared confusing the pool souls), as their home connection. The second guy agreed, then proceeded to tell the first guy to use the VPN.
Then the question/statement came up that we wasn’t sure it was fixed because one of the QA guys testing the issue from another point reported an error. Here is where I nearly lost it, and yelled down the phone, but decided on the mute button instead. The second developer guy, proceeded to tell us that dev guy 1 got the error because his clock was wrong, and that because the clock was wrong, he didn’t see the code changes yet. At this point, I decided to mute the phone, and go find a drink. I was thinking of something incredibly strong, but I have more testing todo tonight as they are also doing load balancer updates.
Right now, I’m trying to figure out if dev guy 2 was joking or not. He’s not the kind of guy to joke about stuff like this, so I’m really worried… I mean really worried.