Could tired pilots make fatal mistakes? How much fatigue is too much? A nervous Congress wanted answers. So human factors researchers at NASA were told to investigate the significance of pilot fatigue in aircraft operations.
These questions could not be safely tested in the real world. Simulation, the researchers decided, was the best medium for assessing pilot performance. They picked a Boeing 737 simulator to examine how well two-person cockpit crews would respond to wicked but realistic scenarios.
The researchers tested two groups of test crews: those who flew the scenario after a minimum of two days' rest, as if it were the first leg of a three-day trip, and those who flew it as the last leg of a three-day trip. As expected, the crews flying the last leg reported significantly more fatigue. But, to the researchers' astonishment, "fatigued" crews were rated as performing significantly better and in fact made fewer serious operational errors than the rested crews.
"In hindsight," the NASA researchers reported, "the finding shouldn't have been a surprise at all." The fatigued crews had just completed three days of operation as a team, while the rested crews did not have the benefit of recent experience with their other crew members.
Did NASA's findings imply that pilot fatigue was irrelevant to flight safety? Of course not. But the simulations uncovered unexpectedly compelling issues. Managing pilot pairings may prove far more important for flight safety than managing pilot hours.
The airlines took note. They now monitor and manage pilot pairings. That this life-or-death management issue was discovered by accident speaks volumes about the challenge of creatively managing surprise.
Louis Pasteur once remarked that "Chance favors the prepared mind." But as the NASA experience illustrates, it's equally true that chance favors the prepared prototype: Models and simulations can and should be media to create surprise and serendipity.
Yet in most organizations, surprise is not welcome. More often than not, companies design prototypes to eliminate surprise, not create it. Models and simulations are usually built to test ideas, not generate new ones. As a result, they are rarely expected to contradict the assumptions that gave them birth.
But when those contradictions inevitably occur, does your organization make use of them, as the airlines did, or does it willfully ignore the model's assumptions? Clark Abt of Abt Associates, a pioneer in bringing simulation to public policy, recalls running a simulation in sustainable economic development for the Agency for Inter national Development (AID). "The simulation was biased in favor of saving the forests," he notes, "while still allowing for a growing population and increasing the standard of living."
The goal, Abt says, was "to learn how to save the environment in a politically responsible way while having healthy economic development." Unfortunately, Abt soon discovered, practically every run of every simulation led to the rapid destruction of the ecologically cherished but commercially irresistible forests.
"By the end of the day, the forests were all gone," Abt remembers. So what did AID do in the face of this consistent-and politically incorrect-outcome? The agency swiftly shut down the exercise. By Abt's and AID's own expectations, the simulation results were an unpleasant shock. The desired outcome-that painful trade-offs between economic and environmental concerns could be avoided-simply wasn't true.
The NASA and AID examples illustrate the power of prototyping and the surprises it generates. It forces people to choose which aspects of surprise they will try to understand and which ones they will deliberately ignore. In the process, they will be forced to rethink their most fundamental assumptions and change their own behaviors. When models and simulations behave in surprising ways, the people who use them have an opportunity to do the same.
Michael Schrage is co-director of the MIT Media Lab's eMarkets Initiative. He is the author of Serious Play: How the World's Best Companies Simulate to Innovate. He can be reached at schrage@media.mit.edu.