New Methods to Measure the Safety of Military Equipment
5
2,009
Guest Commentary By William Edmonds, Jenelle Hirano, and Elizabeth Rodriguez-Johnson
The Defense Department has developed a new tool for acquisition programs that was designed to gauge the safety of weapons systems. It is called the “system safety metrics method” tool, or SSMM.
SSMM can be especially useful for programs that are driven by an urgent need, such as the mine resistant ambush protected vehicle. The model would help to properly identify potential hazards to the operators of the vehicle — including overheating or rollovers.
Other programs that would benefit from SSMM are commercial off-the-shelf procurements such as body armor. The equipment may be effective in protecting against ballistic threats or fragments, but that does not ensure it is safe for the user because a provider may not have tested or documented flammability or ventilation issues.
Safety engineers at the U.S. Army Combat Readiness Center, Ala., see the potential for the model’s practical application to manage safety risks and prevent mishaps.
Identifying safety vulnerabilities early in a program’s life cycle is imperative for protecting troops and reducing preventable accidents. In addition, detecting safety weaknesses early on helps to save costs. The earlier problems are discovered, the better they can be addressed and remedied before the government invests heavily in development, testing and deployment.
The trouble with traditional methods for judging safety programs is that they involve time-consuming and costly empirical studies that are scheduled too late in the timeline to be of greatest use to the program. Most safety evaluations take place no sooner than the Milestone B engineering and manufacturing development phase after a program has been established and resources are provided for safety support. In some cases, important data may not even be collected until after the program ends. Audit results usually are not available immediately, and the analysis may take several months or more to complete. Programs may request an audit during development, but audits are not systematically performed on every program.
Under the traditional evaluations, for example, testing a vehicle’s potential to roll over at a certain speed or turning radius would require the creation of a model of adverse driving conditions, coupled with the vehicle’s operating characteristics. The evaluation may take months or even years. By the time a program analyzes the data, the production may well under way. Defense Department data shows that the majority of mishaps are attributed to human error, which may occur as a result of applying procedural and training controls for hazards that could have been addressed with design solutions.
The Department’s defense safety oversight council created the SSMM through its acquisition and technology programs task force.
The task force came up with a standardized approach to define the measures of system safety practices, similar to the CMMI (capability maturity model integration) model for assessing system design maturity. The group estimated that defense system safety programs were currently at only a CMMI 1 level.
Five characteristics were chosen as the basis for measurement: integrity, maturity, competency, capabilities, and effectiveness. Unlike a traditional safety program evaluation, it relies on one-on-one interviews to determine whether the appropriate resources have been positioned to identify hazards and manage risks. Other questions include: How is acceptance of risk documented and reported? Are well-trained safety personnel involved in the specification and design process? Do the design specifications reflect the impact of safety inputs?
Integrating systems engineering with system safety is paramount for reducing risk. A safety program must ensure that hazards have been appropriately identified and assessed. Also, program officials must understand how a hazard is characterized, what mechanisms will produce its occurrence and the severity of the outcome if it does occur.
The task force developed guidelines to accompany the questions so practitioners experienced in the subject matter can easily bring themselves up to speed on the interviewing techniques. In addition, the questions are available in an electronic format and can be self-administered by program personnel.
The method includes a detailed process for collecting, scoring and analyzing the resulting data. Questions are framed to exclude “yes/no” answers, and interviewers rate the responses according to the model’s five scoring levels. Each interview takes about 90 minutes. Entering data takes about 10 minutes following each interview.
So far, beta test results have provided positive feedback.
The Army Aviation and Missile Command safety Office in Huntsville, Ala., conducted one beta test using 17 program practitioners. The results showed the method is effective. The testers concluded the approach could help reduce both the frequency and severity of mishaps.
Following the beta test, the Safety Engineering and Analysis Center of APT-Research Inc. in Huntsville, sponsored a workshop with 21 participants to discuss potential applications of the model. According to APT-Research’s Pat Clemens, “Although the model cannot predict the number of accidents that will be reduced, it can help programs improve their capacity to produce that reduction through an effective system safety program.”
The SSMM is not mandatory, but it is being made available to all Defense Department programs.
A copy of the report and the model is available
here.
William Edmonds is a safety engineer at the Army Combat Readiness/Safety Center at Fort Rucker, Ala. Jenelle Hirano is an electronics engineer with the Naval Undersea Warfare Center Detachment Pacific, Ford Island, Hawaii. Elizabeth Rodriguez-Johnson is deputy chair of the acquisition and technology programs task force at the office of the deputy undersecretary of defense for acquisition and technology.
