ARTICLE

May 2004

Weapon Evaluators Must Change, Or Risk Irrelevance, Warns Christie

by Sandra I. Erwin

At a time when the military services are rushing to field new equipment to wars in Iraq and Afghanistan, many systems see combat before they go through operational testing. This is one among several reasons why weapon test organizations must adapt to the times or, otherwise, risk becoming irrelevant, said Thomas P. Christie, the Pentagon’s director of operational test and evaluation.

OT&E agencies should not forget that the troops in the field are their primary customers, and it’s important for testers to give users a realistic appraisal of their equipment performance, even if those weapons have not yet gone through full operational testing, Christie said in a recent speech to the National Defense Industrial Association Test and Evaluation symposium, in Reno, Nev.

“When so many of our systems go to war before initial OT&E and before full-rate production, users need up-to-the-minute, continuous test and evaluation to keep them informed of system capabilities and limitations. Even after fielding, the acquisition community needs continuous evaluation to feed spiral development,” Christie said.

In the post-9/11 era, testers are facing “formidable” challenges, he noted. They often are perceived as impediments to progress that slow down the acquisition process. “Program offices and developers appear at times to be learning faster how to avoid testing than we are learning to do it better,” he said. “Right or wrong, the concept of milestone-driven OT&E appears to be becoming a process of the past.

“Either we change our way of doing business, adapt to the new acquisition paradigms and the realities of the war on terrorism, or we will find ourselves becoming irrelevant with dire consequences for our operational forces.”

The Defense Department’s aggressive push for “joint” weapon systems also has implications for testing. Christie’s office is expected this month to produce a “roadmap” that outlines future steps to ensure that test and evaluation help enhance joint capabilities. “We are working with the service and defense agency test communities to satisfy this direction,” Christie said. “There is no effective method for conducting OT&E that cuts across service lines although, in most actual combat environments, the U.S. must conduct combined operations.”

Testers, meanwhile, can expect to see tight budgets. Generally, funds earmarked for OT&E do not have separate line items in the budget, and often are vulnerable to diversion to other purposes, Christie said. “I reject the claims of the many critics of the testing process that overall OT&E costs and schedules are excessive—in fact, they’re a very small part of system costs.”

In the 1990s, when the budgets for testing and testing infrastructure were being slashed by the services, a movement emerged at the Defense Department in support of using modeling and simulation technology to replace some portions of the live testing process. Despite advances in technology, however, Christie believes that modeling and simulation in support of T&E has been a “mixed bag, at best.”

The experience with M&S, overall, has been a “major disappointment of promises undelivered,” he said. Surely, expectations were unreasonable. Although some design problems can be modeled, these tend to be small changes in well-understood designs. Defense systems do not tend to be of this ilk, according to Christie. “When the system technology is cutting edge, its real limits are probably not well understood. You cannot replace testing with modeling in that case.”

Money is another problem. Many program managers would like to finance the development of models with money from testing—trade off testing for modeling. “That timing is off,” Christie said. “Modeling, to be successful, has to start early. Using OT money is too late.

“Defense systems encounter a lot of problems in development—a fact that the OT community is painfully aware, because so many of those problems appear in IOT&E. To overcome these, in the best case, takes additional time and money. The role of modeling should be as something extra that can be done to help the success of the program—not some trade off with testing.”

Budget cuts also have affected the testing ranges and their ability to acquire the latest instrumentation and target technologies. “I am concerned that our T&E infrastructure is not in the best of shape needed to meet the challenges of the future,” Christie said. Because so many programs slip in their schedule, the testing ranges have not been overburdened. “Lord knows what would happen if all the programs that claimed to be ready for testing in 2004 actually showed up for testing.”

Christie rejected the criticism heard within the Defense Department that testing slows programs down. The Army spent 20 years developing the now-cancelled Comanche, and the program still was several years from its OT&E and production decision.

After $36 billion and nearly 20 years in development, the Air Force F-22 fighter is about to enter operational testing. “I challenge you to show me where operational testing has held these programs up or has cost us an arm and a leg as some of our critics would claim,” Christie said.

As they look to the future, he noted, testers should be concerned about the increasing complexity of systems and tactics to be tested, the need for better trained people in the T&E business, the massive amounts of data becoming available and the concomitant requirement for more sophisticated evaluation techniques.

Back To Top