Army and Marine Corps officials have praised simulators because they cost less than live combat training. With the flip of a button, instructors can change settings from desert to jungle, run different scenarios and give instant feedback to troops instead of having to set up elaborate, expensive and time-consuming war games.
However, neither service is able to precisely measure just how much money is being saved, or to what degree simulators are able to improve troop performance, said an August report by the Government Accountability Office.
Part of the problem is that while the Army and Marine Corps have subjective ways of documenting that virtual training improves proficiency, they don’t have an objective method to measure how much it helps prepare troops for combat or live exercises, said Sharon Pickup, GAO director of defense capabilities and management. “The Army needs an institutional approach,” she said.
Both services need to establish metrics to calculate just how effective their simulators are, said the report. Furthermore, they need a more comprehensive method to compare the costs of live and virtual training, it said.
The Army’s program executive office for simulation, training and instrumentation has recognized the need for more information on the value of what it does, said Army Lt. Col. Johnny Powers, PEO STRI’s product manager for simulation to mission command interoperability.
It currently is studying how to best evaluate the usefulness of virtual training — both by using data and subjective measures, he said.
“This study will provide methodologies to help determine the return on investment for the use of simulation tools to provide command and staff training to units,” he told National Defense in an email.
There are many methods the Army uses to measure the value of simulation for individual and crew-level training, but there is no overall methodology to determine if they actually “increase an organization’s collective task proficiency,” Powers said. Currently, “commanders and other leaders subjectively assess unit task proficiency based on published standards and the assessor’s experience level.”
The Army also routinely collects data on how effectively an individual is using a simulation, and a soldier must meet doctrinal standards before moving on to the next phase of virtual or live training, he said.
Marine Corps training is based on its Systems Approach to Training (SAT), Laura Junor, deputy assistant secretary of defense for readiness, said in the official Defense Department response to the report. SAT already includes a policy on developing outcome-oriented performance metrics.
But data on the usage of simulators and subjective assessments from experts are not enough, the GAO report said. The services need to establish indicators to more broadly measure the impact of virtual training on troop performance.
The GAO did not prescribe or suggest what performance metrics the services should use to measure the impact of virtual training. One possibility is collecting data on how a servicemember’s abilities change before and after using a simulator, Pickup said. Doing so would give the services insight on how much simulator use is optimal before an individual or unit reaches a plateau in proficiency gains.
PEO STRI’s study has completed its first phase, which included building a model to determine the service’s current investment. The Army also conducted interviews to develop a survey tool, which will help evaluate the usefulness of virtual training for leaders who conduct training that is supported by simulations, he added.
“Early indications are that … units who use simulations to conduct training increase their performance and that the commanders believe that their units are better prepared to perform their required mission,” Powers said. The first part of the study will be made public after the paper is briefed Dec. 3 at the Interservice/Industry Training, Simulation and Education Conference.
While the first portion of the study focused on brigade combat teams, the next phase will validate the effectiveness of simulations across the range of military operations and types of units, Powers said. The service will collect data from mission training complexes in the continental United States, combat training centers, and the 8th Army stationed in South Korea, he added.
Collecting the data needed for such a comprehensive study has been difficult because many Army units are still focusing on wartime missions. “The data for actual use of the simulation still needs depth since many units were conducting theater-specific training that did not exercise the entire range of military operations or the entire staff,” Powers said.
It is going to take time for the Army and Marine Corps to work simulators into the standard protocols they use to get units ready for combat, said Gene Colabatistto, group president for CAE’s military simulation products, training and services. Compared to the Air Force’s long history of using flight simulators, the Army and Marine Corps have only begun in recent decades to employ ground-combat simulations, he said.
“What they’ve done is they’ve taken the instructional techniques that were used in a live environment, and they’ve just moved them over to the simulators,” he said. “What we’ve learned on the air side is that while some of that is applicable, not all of it is. There are things you can do and you can learn in a simulator that you can’t do in a live environment.”
About 70 percent of CAE’s defense business is for aircraft simulators, he said, but the company has also sold ground trainers to the Army and Marine Corps, including one that teaches Abrams tank maintenance.
Colabatistto suggested that the Army and Marine Corps focus on three areas in order to find savings and make virtual training more effective. The first is the individual’s use of simulation to practice skills such as using a tank to maneuver around or over an obstacle.
The second area would involve analyzing how multiple simulators can be networked to do team-level training of more advanced tactics, such as how to engage a target.
Both services also should look at how best to pull simulations together in training centers that combine classroom instruction, virtual training and after-action reviews, he said. “That is the type of environment that ground combat units have not really had access to at the same level as an air unit.”
Other industry officials believed the GAO report did not comprehensively look at all of the activities the Army and Marine Corps do to test the value of simulators. Clarence Pape, Intelligent Decisions Inc.’s vice president of simulation and training, pointed to the Army’s Bold Quest 2013 exercise that tested whether virtual training improved soldiers’ performance in a live environment.
One of the simulations used in that exercise was Intelligent Decision’s Dismounted Soldier Training System, in which soldiers’ movements are captured by sensors and recreated onscreen in a virtual environment. The system has been used to teach squad level tactics to more than 25,000 soldiers, Pape said.
“We collaborate [with the Army] on metrics that we want to collect with Dismounted Soldier all the time, whether it be the particular effectiveness of a piece of technology or in the evaluation of future-state technology,” he said. “We track the scenarios that the individuals go through. We track the after action review. We track the exercise. There are metrics there.”
The system helps measure how well the squad communicates and operates together, Pape said. The simulation itself can track how many shots it took for an individual to hit the target, for example. More importantly, soldiers are able to receive feedback in after action reviews led by unit leaders, who then can run a similar scenario to see whether those corrections have been made, he said.
Pape pushed back on the idea that such methods of evaluation were subjective.
“Tactical doctrine is tactical doctrine. There’s a certain way that you go through a door or clear a building or check the perimeter for security,” he said. “There are straight tactical events that can be measured. You either do them right or you do them wrong. You either follow tactical doctrine or you do not.”
Establishing performance metrics is only one part of the problem, according to the GAO report. The Army and Marine Corps also need a methodology for comparing the costs of live exercises and simulation.
Both services consider life cycle cost to operate and maintain systems after they are fielded, but they don’t have a way to identify the “universe of costs” of simulators and live training. Some of those expenses aren’t always apparent, officials told the GAO. For instance, ammunition for live training may be more expensive, but if it is bought and isn’t used within its shelf life, the service must pay to demilitarize it.
Evaluating the cost benefits of virtual training is a tricky proposition even in the aviation sector, where simulators have been more widely adopted, Colabatistto said.
When comparing cost per flight hour with the cost per hour in a simulator, the overall rule of thumb is that simulators operate at about 10 percent the cost of live training. Those figures don’t necessarily take into account factors such as the depreciation of aircraft or the effect of accidents during live training, which can cost lives and treasure, he said. CAE has started return of investment studies for its air customers to get a more complete answer to this question.
However, comparing the cost of live and simulated ground combat training presents its own set of challenges, said Colabatistto, a former Marine. “We never [only] took one tank out on the range; you would bring a battalion of tanks on the range. And you really didn’t count your costs by tank or by training hour because you would put those tanks out on the range for 30 days,” he explained. “So the idea of one hour of training — like you log one hour of training in an airplane — is not an apples-to-apples” comparison.
Colabatistto believes that over time the Army and Marines Corps will devise better metrics to understand the cost-benefit of ground simulators. “I think we’re simply at an earlier stage of adoption. They’re becoming familiar with the systems, and for the first time we see them being adopted en masse,” he said.Photo Credit: Marine Corps, A combat convoy simulator