Modernizing Defense Through Crowdsourcing

iStock illustration

To support U.S. competitiveness, policymakers need to make strategic decisions for defense modernization, including where to invest attention and resources. Whether identifying threats or opportunities, leaders benefit from accurate forecasts about the capabilities and limitations of emerging technologies and industries.

Crowdsourced forecasting, or CSF, has received increased attention and advocacy in recent years due to the convenience afforded by widespread internet access combined with positive experimental performance for questions regarding politics and international affairs.

Although further experimentation and evaluation are needed, such forecasting could prove to be a valuable tool for defense modernization planning, technological assessments and related activities in the Defense Department and industry.

Crowdsourced forecasting refers to a variety of systems for eliciting quantitative forecasts from groups of people about defined outcomes. There are two major types of such systems: “prediction markets,” where participants use real or fake currency to trade outcome “contracts” at prices which translate to the market’s probability estimates, and “prediction polls,” where participants state their estimates, and these forecasts are aggregated into simple or weighted averages usually based on past performance.

Some active examples of platforms using prediction polls include Good Judgment Open, INFER Public, Metaculus and the U.K.’s Cosmic Bazaar. Examples of real-money prediction markets include the Iowa Electronic Markets and Kalshi.

The RAND Corp. has developed the Delphi method, which involves cycles of surveying select experts for their estimates and reasoning, has some similarities to crowdsourced forecasting but typically uses a narrower “crowd” and puts less emphasis on evaluating and rewarding individual accuracy.

Researchers have analyzed crowdsourcing for decades, and some early platforms such as the Iowa Electronic Markets have been effective at forecasting elections. However, one of the most influential recent studies was led by the University of Pennsylvania’s Philip Tetlock for Intelligence Advanced Research Projects Agency’s Aggregative Contingent Estimation competition in the 2010s. Tetlock’s project, which birthed Good Judgment Open and was the focus of the book “Superforecasting,” found that teams of skilled volunteer forecasters outperformed intelligence community analysts by sizable margins on a variety of geopolitical questions.

Although there is little experimental evidence comparing crowdsourced forecasting to alternative methods for science and technology forecasting, there are at least three noteworthy arguments for why it might offer improved accuracy over alternative methods.

First, it provides clear accountability or incentives for analytical effort and impartiality, requiring outspoken skeptics or optimists to numerically put their money, or reputation, where their mouth is.

Crowdsourced forecasting also makes it easier to evaluate relative performance and put more weight on better-performing forecasters.

And it can benefit from the “wisdom of the crowd” by allowing people with relevant knowledge and different biases to participate, especially when questions involve knowledge from multiple domains. For example, a forecast of Chinese solar energy usage likely benefits from knowledge about solar panel technology, competing energy technologies and Chinese politics.

Crowdsourcing can be useful even if it does not consistently outperform alternative methods. For example, it can provide a benchmark for evaluating alternative methods such as subject matter experts or machine learning models. It can also support analyst training and talent identification for recruitment, and it could be used in combination with traditional methods to flag where deeper analysis may be warranted to resolve stark differences between methods.

A national science and technology crowdsourced forecasting platform would face many design options and challenges regarding whether to use prediction markets or prediction polls, public or government-only access and user anonymity.

A government-only, open-source prediction polling platform is a potential model. This approach mitigates some concerns with prediction markets, such as insider trading and reduces the risks of adversaries manipulating or learning from the predictions. However, for science and technology topics it seems particularly important to incentivize participation among universities, laboratories and other relevant communities.

Some questions are not well suited to crowdsourcing, including some with long timelines and many that heavily relate to classified information. However, sites such as Metaculus and INFER provide many examples of the kinds of questions that seem appropriate and useful, such as “Will Moore’s Law end by 2025?” and “When will commercial space launch be priced at less than $500 per pound?”

One can also envision questions about China’s semiconductor manufacturing capabilities, non-state actors’ usage of autonomous weapons, commercial battery technology, if/when quantum computing will crack current encryption, the cost and effectiveness of novel explosive compounds and foreign adoption of Chinese telecommunication infrastructure.

Forecasts for such questions could contribute to decisions regarding training topics or wargame design, project budgeting and planning, intelligence collection targeting, export controls, investment in countermeasures or spin-on capabilities and more.

Although crowdsourced forecasting certainly will not be a crystal ball, it could be helpful for making better decisions, including defense modernization planning. Given the significance of these decisions and given the findings of related research by Tetlock and others, further experimentation with such methods appears to have high potential value at a relatively low cost.

Ultimately, if defense planners want better science and technology forecasting, crowdsourced forecasting is a concrete but underused idea for pursuing that goal.

Harrison Durland is a graduate student at Georgetown University and former ETI Research Intern.

Topics: Emerging Technologies, Test and Evaluation, Science and Engineering Technology