Surviving Panic Attacks in IT

Sy Liebergot, a NASA flight controller during the Apollo years, described the famous “Houston, we have a problem” moment of the Apollo 13 flight this way:

It was an incredibly tense time. … failure had occurred and I had no quick answers. As EECOM, I felt very much on the spot. I could feel the chill of panic begin to well up. There was no one to whom I could turn, and I will admit that a fleeting thought of getting up and going home did cross my mind. Of course, that was not an option!

I firmed my grip on the security handles of my console and brought my emotions under control.

— from “Go, Flight” by Rick Houston and Milt Heflin

I cannot tell you how happy I was when I read that.

I have been in IT for over 20 years, a profession one of my colleagues once described as “weeks of tedium interspersed by moments of sheer panic.”

Here’s some advice for any new IT managers out there: no matter how much education you have, no matter how many years of practical experience you’ve amassed, one bright, sunshine-filled day someone will run up to you with a problem that you have no idea how to solve.

The best people in this scenario are the ones who work the problem, one solvable detail at a time. The higher up the chain you are, though, the more you are forced to stop and look at the big picture, theorizing all the ramifications of the issue in front of you. I’ve found the earlier I start thinking about work-arounds and contingency situations, the better off I am if the initial fix does not work.

But there are times where looking ahead can overwhelm you. The sheer enormity of the problem can throttle you with panic.

About 15 years ago, I had just taken over as IT Manager at a printing company when I got word that the computer on our 3-million dollar press would not boot up. I had never worked with that computer before. We had no manuals. The only backup of the settings was on a floppy drive that the machine refused to read. The software was so obscure that even Googling it got us nowhere. And it ran on an ancient version of DOS loaded with German-language-based software (which I will forever think of as “das DOS”).

It was a nightmare.

I took one look at it and knew I had no way of getting it back on its feet. I simply didn’t have the tools. That didn’t stop us from trying a million solutions, including replacing every piece of hardware inside the machine until we’d created this strange Franken-computer doppelganger of the original.

And every 15 minutes or so, one of the upper managers came by to get a status update and remind us that the production output of the entire company was halved until we could get this back online.

In the end, we had to track down another version of this exact, obscure, late-80s machine and have it shipped overnight air from Europe. Then we hired a team of people to fly in and reset the entire system (and, of course, create a working backup for us).

The problem was solved, but the entire ordeal took over a week, during which the financial guys were going nuts.

Now here’s the little secret I’ve never told anyone:

I had a period of sheer panic at the start of this. My blood pressure shot right up. I stood at the console, realizing this was now my problem and that I had never been trained on it, had no idea what the original configuration looked like, and did not possess spares of any of the specialized hardware that connected the machine to the press. Everyone in the room was shouting at once.

That’s when I got this very clear image of myself running out the door. No coat, no purse, no car. In my mind, I was just hauling ass down the road.

It was tempting.

I think I visualized that for about 20 seconds while pretending to read something on the monitor. Then I promised myself I’d concentrate on one problem at a time and see what I could do to make things better. That got me through the first hour, and the next time panic seized me, I pictured how far away I’d have been right then if I had started running when I wanted to. The idea made me smile and I got back to work.

A decade and a half later, I have dozens of similar war stories to tell. Glitches that took down servers. Upgrades that caused 10 problems for every issue they fixed. File catastrophes caused by users who should have known better.

It gets easier.

But through it all, I occasionally still have to stop and focus on alternate-reality me, still out there running, Forrest Gump style, across the country.