A Pentagon-funded survey of major defense programs revealed what
many had suspected all along: that simulation-based acquisition
is more of a myth than a reality.
Simulation-based acquisition is a buzzword that illustrates the
notion that weapon systems can be designed and developed entirely
as digital models and that those models are accepted as valid prototypes.
The sponsor of the survey was James F. O’Bryon, deputy director
of operational test and evaluation at the office of the secretary
of defense. The study covered 22 major defense acquisition programs.
Each program office was asked whether it used simulation-based acquisition
methods and, if so, to explain how it was implemented. The only
program that did not provide data was the Joint Strike Fighter,
because it’s still in a competition, so any information on
JSF models was considered proprietary. Ironically, the JSF program
is among those that most heavily have used modeling and simulation
in the design and development phases.
Among the questions asked were how digital models are organized,
who pays for the models and whether the models are validated and
accredited.
“To ensure candor from the program managers, we agreed to
not release information on specific programs, but rather service-wide,”
said O’Bryon in an interview.
The survey, he said, “paints a rather bleak picture.”
Even though the Defense Department spends millions of dollars each
year developing digital models for various applications, simulation-based
acquisition is not pursued in any organized manner, O’Bryon
said. “We really don’t have SBA. A lot of money is spent
on models and simulation. But it’s very disjointed. In some
cases, it’s spent multiple times on similar models,”
he said. “Most acquisition programs do not have modeling and
simulation plans.”
Industry executives either are being disingenuous or are fooling
themselves when they say that SBA is “here and now,”
said O’Bryon.
During an August conference on testing and training in Orlando,
Fla., the president of Lockheed Martin Missiles and Fire Control,
Stan Arthur, said that simulation-based acquisition “has arrived.”
That is false, said O’Bryon. “SBA is here in name.
It’s only a slogan ... A bumper sticker. ... It does not exist.”
Many people in the industry, he added, “want to declare success.
It makes the industry people look good, because they don’t
spend money testing” the systems.
At the Defense Department, he said, “We are using simulation.
But we don’t have a simulation-based acquisition system, using
simulation systematically. If that were the case, O’Bryon
said, huge financial savings could be made.
“We are spending billions of dollars on models and it’s
not very well organized. ... On the training side, they are more
organized than on the acquisition side.” Models are used,
for example, to replicate the terrain where a weapon system will
operate, the atmospheric conditions, the radar signature, among
others.
According to the survey, more than 80 percent of the models used
in the 22 programs were unique to that project, and not shared with
others. In many cases, the models are owned by contractors, not
by the government.
The problem, said O’Bryon, is that program managers don’t
have an obvious motivation to invest in modeling and simulation.
Most of them are in the job for two or three years, which means
that, by the time the models and simulation work is completed, they
have moved on to other assignments. “Program managers have
a high turnover rate, so they don’t want to invest that money,
because it will not benefit them in the short term,” he said.
Sometimes, program managers prefer to not have realistic models,
because they can “make the program look worse,” said
O’Bryon. The performance of a weapon can deteriorate when
factors such as turbulence, terrain or targets are factored in.
If the models are “too realistic,” said O’Bryon,
“the estimates look worse.”
Rarely do models help a program “look better,” he added.
“Realistic models are not always viewed as a positive thing.”
Reform Plan
O’Bryon’s office has submitted a proposal to the undersecretary
of defense for acquisition, technology and logistics, which would
help promote the use of SBA. The plan is called MASTER (modeling
and simulation, test and evaluation reform).
The plan is built on the premise that there is “virtually
no new money” for modeling and simulation. The funds would
have to be taken from other areas within program budgets. The program
managers, said O’Bryon, “control most of the money that
could be spent on modeling and simulation.”
O’Bryon is proposing the creation of a “modeling and
simulation consortium,” which would be funded by “negotiable
arrangements with program managers, who would contribute to the
consortium.” The consortium would help the program managers
advance their projects through the use of SBA and would encourage
them to share models, when possible.
“Program managers typically need modeling support,”
said O’Bryon. Most frequently, he said, they need models to
simulate missile or aircraft aerodynamics, atmospheric conditions,
target signatures and vulnerability.
Under the current way of doing business, he said, the program manager
typically pays for models that already have been paid for by someone
else. “The consortium would help save money and duplication,”
said O’Bryon.
The government agency that would oversee the consortium, he said,
is the Defense Modeling and Simulation Office. “DMSO would
maintain and chair the consortium,” said O’Bryon. Currently,
“DMSO can’t do its job because the program managers
control the money. There needs to be a revision of the acquisition
policy.”
O’Bryon, however, does not advocate the use of modeling and
simulation as a substitute for live testing. “Modeling and
simulation [technology] has been sold as a cost saver. We can’t
substitute models for testing, because we don’t know how to
do models yet.”
The current acquisition regulations require a modeling and simulation
plan for every program, he said. “The money in SBA today is
in stovepipes. They don’t share” the technology. “They
have a disincentive to share, because it’s their neck. It’s
not their fault. It’s the system.”
O’Bryon noted that there is a law—which is not often
enforced—that mandates congressional approval for any weapon
system-related model that costs more than $50,000. “It’s
not being complied with,” he said. “They are not tracking
the law.”
Currently, there are three pilot programs in place that will be
used to determine whether SBA can work: the Air Force T-38 trainer
aircraft avionics upgrade, the Army’s common missile and the
Navy extended range guided munition.
Frank Cappuccio, a Lockheed Martin vice president who works on
the Joint Strike Fighter program, agreed that there is “a
lot of rhetoric” about simulation-based acquisition. However,
he said, “the savings are there.”
During an SBA conference sponsored by the National Defense Industrial
Association, Cappuccio said that the Joint Strike Fighter program
achieved a 50 percent savings in “engineering, acquisition
life-cycle time and cost.” The savings, he said, “were
incorporated into our [JSF] proposal bid.”
The implementation of SBA, he cautioned, is a process with “numerous
minefields.” The biggest obstacle, he said, comes from the
financial department. “SBA will test everyone’s resolve,”
said Cappuccio. “If you have the resolve, you can convince
your finance people to pass the savings to the government.”
For SBA to work, he said, “it takes a consistent vision from
the government.” In the JSF program, that was possible because
the program manager, Marine Maj. Gen. Michael Hough, is an advocate
of the use of simulation and made sure it was implemented.
“A good vision is not sufficient,” said Cappuccio.
“You need pressure for change. ... This is something you have
to do to be competitive.”
Government leaders must decide, he said, whether the replacement
of hardware testing with simulations ever will be accepted.
“Which company will take the first step to tell the government
to not shoot an aircraft with a 20 mm cannon, that I am going to
do it with modeling and simulation?” Cappuccio asked the audience
at the conference. “It’s not enough for program managers
to champion SBA.”
In the JSF program, both contractors—Lockheed Martin and
Boeing—had an incentive to use modeling and simulation to
cut costs, because it would make their proposals more competitive.
But unless there is a competition, said Cappuccio, there are “no
financial rewards for industry to cut costs. ... In something that
is not a competition, it would be a smaller bill and higher risk.”
For SBA to take hold, he said, “the government has to be
a partner” and accept that models can offer legitimate testing
tools.
Another issue the government must address, he said, is how to protect
industry’s proprietary data. “There is no trust in our
industry,” he said, because “government people give
away the data in competitions.”
There is an “entrenched belief in the financial community
that proprietary data is what makes a company unique,” said
Cappuccio. The JSF school of thought is different, because “we
give up proprietary data” to the government.
It’s an emotional issue to those who believe that if the
government takes proprietary data, “it is going to take away
from the identity of the company.” However, “we do it
for national security reasons.”
In those programs that require interoperability, “it would
make sense to develop a standard for SBA. Industry would rally to
it quickly,” said Cappuccio. “We have done it before,
for avionics standards.”
The 1553 data bus standard was developed, he noted, “because
every company was fed up with incompatibility. ... When it’s
in your best financial interest to share data and have common standards,
you’ll pull it off.”
Philip W. Cheney, Raytheon vice president of engineering, said
that SBA does not offer any significant return on investment. “It’s
a continuing squeeze on company resources,” he told the conference.
“SBA implementation is unique within each service, and at
times, within individual program offices.” The defense industry,
he said, “lacks the ability and incentive to share proprietary
data.”
Air Force Lt. Gen. Leslie Kenne, commander of the Electronic Systems
Center, said during the conference that she is a “zealot”
about SBA. “If we don’t really invest in it, we are
not going to get where we need to be.”