I went to Navy Boot Camp in San Diego, and during that time we had a week of firefighting training so that we could fight fires aboard ship.
The first thing we did was watch a film about a disaster that occurred aboard the USS Forrestal off of the coast of Vietnam in 1967 where a rocket accidentally fired across the flight deck into a group of fueled and armed aircraft. In the first five minutes of the fire, the skilled firefighting crew were killed by bombs cooking off and by the end of the fire fight 134 sailors had lost their lives. It took two days to finally put out the fire and the Forrestal spent a year in dry-dock being repaired and was known from then on in the fleet as the USS Forest Fire.
The Navy studied the flight-deck video from this tragic accident and changed their training so that fire-fighting was a regular part of Navy life. This experience had a deep impression on me, as I still remember much of what I learned even though I have been out of the service for more than 30 years.
The analogy for software projects is that from time to time a project will get out of control. The project will be burning through money, time, talent and resources in a reckless manner with little to show for the effort. If this sounds familiar, then you may be fighting a fire. The first lesson of technical fire-fighting is to recognize that you are fighting a fire, and shift gears to fight the fire. The longer that one delays in admitting that the project is on fire, the more damage is done. When I admit that I am fighting a fire it is easier for me to switch gears into "fire-fighting mode" - to clear-away the red tape so the team can get on with the business of fighting the fire. A good lesson to remember is that when the ship is engulfed in flame we deal with the fire and solve the problem. Lessons learned will happen later, the immediate need is to pull together, create a plan and begin to attack the problems systematically.
Oftentimes, calling in an outside advisor with software experience is useful. They can bring a fresh set of eyes to check our assumptions and help us to see the forest for the trees. Stepping back (even if briefly) can help one to understand if we are chasing symptoms or attacking the core problems.
In Navy fire fighting training I learned to fight the fire at it's base with the right tool. CO2 or AFFF is used on fuel fires, not a 3 inch water hose because the 3 inch hose will cause a fuel fire to explode! The same two principles are true for technical fire-fights - attack the problem at the base with the right tools. Identify the core of the problem - is there a problem with requirements? A design assumption that proved to be wrong? Poor implementation or testing? Once the core problem has been identified then attack the problem with the right tools - What tests can be run to prove to us that we are attacking the fire at the base and not battling symptoms. Isolating the problems with a test framework, or database tuning or log file analysis can often be useful tools to help the team to be effective fire fighters.
The third area to consider is the technical team and their skill set. Each team has its own strengths, weaknesses and biases. It is natural that a team will fight fires with the tools they know, and to work diligently at the solution. There are two things to be learned from evaluating the team - do we know the right tools (or is there a tool we should pick up and use), and does our team have an internal bias that is feeding the fire? In many instances I have seen these two factors exacerbate an already difficult situation, and so I find that the answers to these questions help to navigate the best path forward from a difficult situation.
What is your experience of technical Fire-Fighting? What are the guidelines that you follow when fighting technical fires?