Conventional wisdom says that military forces increasingly will train on simulators because live exercises are too costly and time consuming. But beyond that, there is little consensus on other factors that may explain how such technologies are benefiting the military. The Defense Department has no consistent standards to measure the performance and the benefits of simulation-based training. Many buying decisions are based more on marketing than on empirical evidence, experts say.
“Everyone keeps talking about effectiveness and so-called assessment. But there’s very little of it done,” says Ben Sawyer, co-founder of Digitalmill Inc., a Portland, Maine-based technology consulting firm. “There’s a reason why: Because no one’s willing to pay for it.”
There has been plenty of anecdotal evidence that virtual trainers are preparing troops adequately for war. But concrete data of their performance is lacking.
The ultimate methodology for conducting such an evaluation would be to send observation teams to see how troops perform on the job after receiving simulation training. “That’s the hardest thing to do. It’s very expensive and time consuming,” says John Morey, senior scientist at Dynamics Research Corp., which developed a training technology that includes a system for evaluating a battle staff’s team performance. The company last year was asked to consider conducting a long-term study of an Army brigade that had trained on its technology.
“If provided with the opportunity and resources to do it, I think that would be very worthwhile, and it would potentially yield some exceptional data and results,” says Mike Tremlett, program manager of the Digital Battle Command Team Training program.
Many industry representatives indicated that it would not be difficult to measure the performance of simulators, but that it would be costly.
“What we need to really get these metrics is government support,” says Damian Szigeti, engineering manager at American Systems. “It is expensive to prove efficiency, and it may be easier to use less empirical data and use instructors and things like that to determine how efficient a system is and whether the cost savings are a benefit to the government. But it would be good to have empirical data too.”
The government does have a verification, validation and accreditation process for vetting all training systems before they are approved for use by the military. But the standards can vary, depending on the type of simulator and missions.
“The whole process of V-V and A and metrics is an area that I have felt for a long time is poorly understood and not well applied,” says Mark Herman, vice president for modeling and simulation, war-gaming and analysis at Booz Allen Hamilton.
There are some areas where the effectiveness of virtual trainers is undeniable.
Training a soldier how to fire a weapon or to fly an aircraft are tasks that are routinely accomplished by simulators. The metrics there are simple — how many rounds it takes to hit a target or how long it takes to locate and destroy a target can be compared to performance on a live fire range or up in the skies on a check ride.
In the 1980s, a study compared the performance of soldiers who were trained in gunnery skills at live firing ranges or in simulations. Crews who learned how to shoot through the simulations outperformed those trained on actual systems at the range. The findings convinced many Pentagon officials that simulators were beneficial, says Frank Delisle, vice president of engineering and technology at L-3 Link Simulation and Training.
In flight training, the measure of effectiveness for units that use simulations is always in the number and quality of students that graduate, says Steve Davis, training support manager for CAE USA, whose C-130 simulators prepare pilots at Little Rock Air Force Base. “Flight simulators have gone through rigorous performance testing,” says Herman. “Pilots who have had simulated training are safer than those pilots who haven’t had it. It’s a tried-and-true type of trainer.” Flight simulators also are touted as providing significant cost savings because students do not have to burn fuel and cause wear and tear on the actual aircraft. In the military, a flight simulator can provide 35 training sorties to a squadron every week, whereas an average aircraft may fly only two sorties per day, or only 14 per week, points out Davis. But there are other factors that potentially could be measured. For example, because so many training hours have been moved off the flight line and into simulators, the C-130J training unit at Little Rock operates only eight aircraft — about half the size of traditional Air Force training squadrons.
“The biggest enabler in all of this has been the fidelity of the simulation,” says Delisle.
Everette L. Roper Jr., program manager at Computer Sciences Corp. for the Army’s Flight School XXI, agrees. In flight simulators from the early 1990s, a tree was graphically represented by a green triangle. “Today, you can hover up to the tree and watch the leaf blow from your rotor wash,” he says. “It makes a tremendous difference in the things that you can do in the simulator that you couldn’t do just a few short years ago.”
Troops can learn more tasks because they are immersed in environments that more accurately replicate the real world.
“That helps with this difficult issue of training effectiveness because now you have a more realistic environment so you don’t have to be as subjective,” says Delisle.
But as students progress from the individual-based learning environment to a larger group-oriented exercise, ascertaining the effectiveness of such training becomes much more difficult because the scenarios tend to test cognitive decision-making and judgment rather than skills.
“It’s difficult to measure the effectiveness of that,” says Roper.
Linking simulators together for collective training — called distributed mission operations — is becoming ever more commonplace as the military seeks better ways to prepare forces for joint and multinational operations. Aircrews in disparate locations can take off in a KC-135 simulator and a C-17 simulator and “fly” a refueling mission together. “To be able to practice that mission with the crews who will actually fly … is a tremendous force multiplier for the services,” says Davis. But there are technical hurdles, including limited bandwidth over which to network the systems.
There also are compatibility issues between simulators manufactured by different companies. For instance, the KC-135 crew may look out of its cockpit and see a very different visual representation of the environment than the C-17 crew.
That means an exercise may have to run on lower fidelity data, which can compromise the quality of the training, says Delisle.
“The technology has not matured enough yet to really provide that kind of virtual realism,” he says.
The Air Force is working on common databases so that both aircrews will eventually see the same graphical data, says Davis.
Air and naval forces have simulators that are considered the gold standard. But ground forces don’t often have access to the same level of fidelity, says Army Col. Mark Armstrong, director of the training technology development group at U.S. Joint Forces Command. The group is trying to figure out how to enhance ground-combat simulations so that they can be on a par with those in the air and sea services.
There is also the challenge of training troops for operations in urban environments on existing and outdated simulators. Many of the simulations are unable to replicate the thousands of vehicles and people necessary to help train troops for missions in Iraq and Afghanistan. The Defense Department’s call for more realistic and relevant training is indicative of this problem. “What we’re seeing is a sense of urgency,” says Delisle. “They want to get that capability into the field as soon as possible because they can’t simulate that today and do effective training.”
The ongoing wars have shown a necessity to hone skills in non-kinetic types of operations — nation-building, politics and socio-economic areas such as language and culture — arenas that the military traditionally has not had to worry about.
“We have major gaps in the systems we have right now,” says Armstrong. Industry is working with the command to start bringing in those technologies, such as crowd modeling behavior, he adds. Defense department officials would like to see simulations for these non-kinetic missions that are as sophisticated as those they have for kinetic operations.
“We’ve had lots of models we’ve looked at. But they haven’t proven to be as successful as we would want it,” says Armstrong.
Because simulators often are reserved by units for exercises, constructive models — computers that can replicate those simulators’ functions — increasingly are being used as substitutes.
One of the growth industries in this area is generating video feeds that are streamed from surveillance aircraft, such as the Predator, says Armstrong. An infantry unit learning how to task and use such systems from home station doesn’t need a live or even virtual Predator to train with.
It can benefit from a constructive simulation that models the Predator’s sensors and have that video and imagery piped in directly to their trainers.
To meet the demands for increased realism, companies have brought in technologies from the gaming industry, which has developed artificial intelligence and physics-based systems at a faster pace. L-3 Link is incorporating one of those technologies — an urban complex physics-based environment — into its latest Predator training system. Air Force operators had complained that in the trainer, some of the scenarios were too easy because the target was driving on roads with few cars. In real-world operations, suspects drive through highly congested areas making it difficult to track them. Having the same type of traffic in the trainer would help in honing those skills for combat, they say.
Some experts claim that the debate within industry has moved beyond the effectiveness issue and that there should be more discussion about the proportion of live to virtual training.
“The real debate becomes, what’s the right balance,” says Roper.
At Joint Forces Command, officials are trying to ascertain the amounts of live, virtual and constructive training that is appropriate for various military units.
“It’s very difficult to determine for the USS Enterprise how many virtual steaming hours replaces a seven-day exercise at sea,” says Armstrong. Reproducing that virtually and tying the whole training piece together is something that the naval and air services are wrestling with.
In the meantime, officials also are trying to figure out the future of home station training for ground forces. The command is looking at different connectivity technologies to allow more units to plug into joint simulation training from their military bases.
In times of uncertain budgets and fluctuating fuel prices, the argument for training in simulators is only growing stronger, industry sources say.
“At a time when budgets are shrinking, especially for recurring training and initial qualification training, you need to move those out of the high cost platforms and into lower cost, higher availability platforms,” says Davis.
And that is all the more reason to start assessing the value of virtual training.
“There hasn’t been a real open and honest dialog about effectiveness,” says Sawyer. “Until there’s a much larger and much broader consensus over these types of things, I don’t think this problem will ever feel like it’s going away.”