Defense Department’s AI Posture ‘Challenged’

Illustration: Getty

While artificial intelligence has become a top priority within the Pentagon, a new report by the RAND Corp. has found that the Defense Department has shortcomings in its AI posture.

“Although we see some positive signs, our assessment is that DoD’s posture in AI is significantly challenged across all dimensions of our assessment,” said the report titled, “The Department of Defense Posture for Artificial Intelligence: Assessment and Recommendations.”

The December report was commissioned by the Defense Department’s Joint Artificial Intelligence Center, or JAIC, an organization that was stood up in 2018 to coalesce the Pentagon’s disparate AI projects.

The report focused on six dimensions for its analysis: organization, advancement, adoption, innovation, data and talent. In five of those areas — organization, advancement, adoption, data and talent — the report found significant issues.

The Defense Department’s “AI strategy lacks baselines and metrics to meaningfully assess progress toward its vision,” the study said.

The Pentagon has also failed to provide the JAIC with the “visibility, authorities and resource commitments” it needs, which has made it difficult for the center to reach its full potential, the document added.

In terms of advancement and adoption, the current state of AI verification, validation, testing and evaluation is “nowhere close” to ensuring the performance and safety of AI applications, the report said.

Validation and verification enable the designers of a system to trust its design, while test and evaluation allows managers to assess whether a system meets specified requirements and enables other stakeholders, such as users and operators, to establish trust in it, the study noted.

The Pentagon also faces numerous challenges with insufficient data management. Currently, there is a “lack of traceability, understandability, access and interoperability of data collected by different systems,” the document said.

RAND also pointed out that the Defense Department does not have clear mechanisms for growing, tracking and cultivating AI talent, even as it faces steep competition.

The authors offered 11 recommendations, both strategic and tactical, to address the department’s most critical challenges.

These include a call for the Pentagon to adopt an AI governance structure that aligns authorities and resources with its mission of scaling artificial intelligence across the department. It also recommended the Joint Artificial Intelligence Center, as well as service-led AI organizations, develop a five-year strategic roadmap that would be backed by baselines and metrics.

“A sequence of five-year strategic plans ultimately will be required to achieve DoD’s vision of harnessing the potential of AI to its full extent, as mission-support and operational AI mature to allow use at scale,” the report said. “However, the starting point is now, and it requires the development of the first five-year strategic roadmap with an eye toward what might follow.”

Additionally, the JAIC — working in partnership with the undersecretary of defense for research and engineering, undersecretary of defense for acquisition and sustainment, the chairman of the Joint Chiefs of Staff and service officials — should pursue an annual or biannual portfolio review of Pentagon-wide investments in artificial intelligence.

Officials should also consider organizing a technical workshop, annually or biannually, which showcases the Defense Department’s many AI programs, the report said.
RAND put a premium on verification, validation, test and evaluation in AI, recommending that the DoD work closely with industry and academia in this area. Further, all Pentagon AI efforts should include funding for such activities.

There is also a need for all agencies within the Defense Department to work more closely with AI researchers, developers and operators, and the report called for the creation of mechanisms that would strengthen those connections.

Data is also a key area of reform, and the Pentagon should recognize data as a critical resource, the report said.

The department should continue instituting practices for its collection and curation and increase sharing while resolving issues in protecting the data, the authors noted.

Additionally, the Defense Department’s chief data officer should make a selection of Pentagon data sets available to the AI community which could serve as the catalyst for innovation and enhance engagement with the military.

“Data are critical resources and are not currently leveraged to their full potential in DoD,” the document said. “Remedying that requires DoD to institute processes, practices and standards that encourage, if not require, the collection of data at every possible opportunity and that guide their preparation and curation.”

The final recommendation focused on the need to develop and retain skilled workers, particularly as the Defense Department faces tough competition from the commercial sector.

“Regardless of the specific strategic roadmap taken, DoD’s ability to scale AI will depend on its ability to consistently attract the right mix of AI talent in an extremely competitive AI market,” the report said.

When speaking with experts, developers and program managers, interviewees said they expected to change jobs or roles every two to four years, the document noted.

“This competitive job market and the expectations of participants in it have forced various organizations to contend with a new reality,” the report said.

Traditionally, Defense Department career paths are longer and well-defined, the report said. However, the Pentagon “should neither expect nor necessarily want talent to remain in place for their entire careers, as this is inconsistent with the realities of the AI talent market and the current fast rate of technical advances in the field.”

Topics: Robotics, Robotics and Autonomous Systems