Too scared to act

In an organizational setting, it's often referred to "analysis paralysis".

Last week we had a database server crash twice. This server provides the company's most critical information system. Two-thirds of the company relies on this server to do its daily tasks. Because it was so critical to the daily grind of the company, the previous caretakers decided that a "hands off" approach was most appropriate for this system.

It crashed Monday evening of last week. We spent Tuesday doing a postmortem and identified the likely culprit of the crash. We spent the rest of the week "discussing" what to do about the problem. It crashed again on Friday evening. Both crashes seemed to have been caused by the same problem. So, while we were arguing about how to solve the problem, the problem reached up and bit us again.

The major complicating factor is that this machine is woefully behind on patches. The OS is 5 years old with very few patches installed (and no "service packs"). The database is at least 8 revs behind current and the application framework was old when it was installed 4 years ago.

Trying to bring all of this up to date will most likely break the application that the server runs.

We now have telemetry coming from the server that we think will tell us when the error condition is getting close. This will help us limp along on this machine until we can get a replacement up and running.

But this brings up some good questions:
1. Why isn't this business-critical system running in a more robust architecture?
2. Why was this system neglected for so long?
3. Why hasn't there been a major effort to stabilize this system?
4. Why doesn't the company have dedicated team of DBAs, developers, and sys-admins to keep this system running smooth?

Yeah, the company can come up with reasonable sounding answers to rationalize away these questions, but it still leaves us with a fragile mission critical info system.

Frustrating.