According to some defense contractors, a closer and more direct
working relationship between the operational testers and industry
is warranted and necessary. While we agree with that premise, we
still do not believe, as some contractors have said, that operational
testing is the “long pole in the tent” in fielding a
system, and that giving industry more access to program documentation
will, in some manner, significantly improve the acquisition process.
While some small increase in efficiencies might be possible, we
suggest that industry look elsewhere for significant gains. Currently,
Navy operational testing averages less than 1 percent of the total
program cost and takes less than 7 percent of program development
time (assuming five-year fielding effort). By these measures, the
Navy operational test process is efficient and effective.
The Reality
The acquisition process will not be noticeably shorter or cheaper
with industry observation of testing and access to documents. What
is needed is a shift in the pervasive mindset within the acquisition
community that:
What It Will Take
If we’re to achieve “faster, better and cheaper”
acquisition, we need more awareness within the acquisition community
of the following issues.
1. Sometimes you just can’t get there from here. At some
point, preferably sooner than later, something may preclude a system
from achieving its required capabilities. This could be cost, schedule
or a limitation in current technology. When this occurs, admit the
reality of the situation and concede the effort. Invest the remaining
resources in areas that offer greater promise of success.
2. Some technologies need to mature before they’re adaptable
for military use. The inherent immaturity of leading edge technologies
often makes them unsuitable for use in a military environment (for
example, shipboard, foxhole, etc.). They have their place in demonstrations
and experiments, but not in combat. Some are unstable, unproven,
or just too cost-prohibitive for widespread military use.
3. Commercial products don’t always translate easily into
military environments. Off-the-shelf products were not designed
for combat, but for work in your home or office. Modifying COTS
can lead away from interoperability, and too often has resulted
in dysfunctional C4I and weapon systems. The marketing technique
of proclaiming, “It’s COTS, so it doesn’t need
testing,” is irresponsible risk management.
4. A schedule can drive a program into the ground. Schedules are
important, but they should not be the driving force of a program.
Schedules are tools better used to determine efficiency, rather
than when a system is ready for use. System development metrics
need to be based on performance or achievement, not on the calendar.
(We will accept that program managers have a real challenge here,
because their funding is calendar-based.)
5. Betting a program on a single test is poor risk management.
Would you try to graduate from a university with a degree in engineering
by taking just one comprehensive final exam at the end of your four
years? Unfortunately, some program managers try a variation of this
idea when they reduce or just bypass the opportunity for assessments
by operational testers, and place all their chips on the line in
a single operational evaluation. Our experience in this regard has
reconfirmed. Hope is not a strategy for success.
6. Operational testing is the user’s quality assurance process.
End-of-the-line quality assurance is a poor production practice.
Too often, operational testers are excluded from participating in
developmental testing events and program reviews. Despite the clamor
to “get the operational-testing community involved early,”
there is significant resistance to this concept. Many of the “traditional”
excuses are still heard, such as, “It’s too early, and
the system still has problems. ... If they see something they’ll
tell everyone. ... Acquisition decisions don’t affect operational
testing.”
7. Why can’t industry have access to testing documents? We
would ask the same question about developmental test documents (acquisition
strategies, program baseline agreements, developmental test plans,
data, and reports). The operational testing director needs developmental
testing and industry test plans, data, and reports to plan for efficient,
non-redundant tests and to capitalize on lessons learned.
8. Knowing and following the rules is a sure way to succeed. We
have a bounty of directives, regulations, instructions, policies,
and procedures that have been established for acquisition and testing.
The operational tester is dependent upon two of the fundamental
items: an operational requirements document and a test and evaluation
master plan. Both are requisite documents for any program, and both
essential to conduct operational testing. A disciplined following
of the guidance for acquisition and testing is critical to your
success.
9. Operational testers test based on requirements, not on contract
specifications. When industry is provided the specifications for
a system, or the government releases a request for proposal, the
operational requirements document (ORD) or concept of operations
document (COOD) should also be provided. The requirements sponsor
develops the ORD, and under the new Defense Department acquisition
regulations, an ORD may not exist early in the program. In this
case, a COOD (based on the mission needs statement) will be the
only document describing (albeit at a fairly high level) how the
system will be used. The user, the Navy system developer and the
operational tester should develop the COOD jointly. The resource/requirements
sponsor can then use the COOD to develop the ORD.
10. The key word in operational test and evaluation is “operational.”
In the intended environment, against the projected threat, employing
typical maintainers and operators is how operational is defined.
We are often asked, “How are you going to test the system?”
Our response is that we will test it the way the fleet will use
it, in an end-to-end mission scenario.
Navy Position
The “value added” by controlled involvement of industry
in operational testing outweighs the detriments. Observations of
the testing by qualified industry representatives could produce
benefit, in several areas. First, the observers could put into context
any problems discovered during the test. They would see firsthand
what worked and what didn’t. They could provide feedback to
the program office as well as their company, with an insight that
they previously lacked.
How they will be able to observe testing where only an operator
and his system are present, such as in aircraft tests, will require
further exploration.
We are currently developing a standard operating policy that will
define how industry representatives will be allowed to observe our
operational testing. It will include a requirement for the representative
to execute a non-interference agreement, precluding any interaction
with test personnel or equipment unless specifically requested by
the operational test director. The observer will be allowed to take
notes and will be provided data, with the program office permission,
to analyze failures if they occur.
Access to documentation is another area where the benefits outweigh
perceived risks. Access to the ORD is essential if industry is to
understand what the war-fighter needs. The requirements—not
the contract vehicles and specifications—are used by the operational
testers as a measuring tool. When a mission need statement is translated
to an ORD, and an ORD is translated to a contract specification,
“things” can get lost.
Contractor access to the approved test and evaluation master plan
(with contractual or financial information redacted) is sensible.
The test and evaluation master plan is a program office document,
however; and its control is their responsibility and prerogative.
Access to approved operational test plans makes sense too. Our standard
procedure is to offer the program manager a brief on the test plan
after it has been approved, and the contractor might find benefit
in attending. For some reason, our experience has been that program
managers generally decline this brief.
Industry observer participation in integrated product teams is
also an issue not in the control of the Navy operational testing
community. Program managers charter IPTs, and they or their empowered
representative chair them. We are invited participants and have
no control or influence in who is allowed to attend, observe, or
participate. It seems reasonable to include industry representatives
to comply with Defense Department and Navy acquisition reform initiatives
of partnering with industry.
With regard to providing early test data to industry, the current
procedure for the Navy is to provide the program manager, as expeditiously
as possible, all data relating to a system failure or anomaly discovered
during operational testing. We accomplished this by sending an anomaly
message from the commander of the operational test and evaluation
force to the program manager. The program office restricts us from
interfacing directly with industry developers. This is to prevent
the possibility of perceived tasking to correct or investigate the
cause of an anomaly. Direct operational tester feedback to industry
developers might be misconstrued that the tester is setting a requirement
for the system through informal discussions. We do not want to be
placed in a position of defending a casual “It would be nice
if the system could ...” remark that the developer mistakenly
construes as a requirement to pass testing.
Industry, program managers and operational testers all can benefit
from open communication—but we must be realistic in our expectations
of improvements in quality, economy and efficiency.
Rear Adm. Robert E. Besal is the commander of the Navy Operational
Test and Evaluation Force, in Norfolk, Va. Steven K. Whitehead is
the organization’s technical advisor.