Data Glut Forces National Geospatial-Intelligence Agency to Transform

The word “open” is not often used to describe U.S. intelligence services.

But that is the buzzword at the National Geospatial-Intelligence Agency since its new director Robert Cardillo took over in 2014.

“Openness” has two meanings at the agency which is best known for its thousands of analysts who pour over imagery collected by spy aircraft and satellites in order to determine the intent of rivals.

One is sharing with the world some of the data its collects. Cardillo gave two examples of this at the GEOINT conference in Washington, D.C.

In 2014 during the worst of the Ebola outbreak in West Africa, he instructed his staff to create a website where medical personnel could see maps where outbreaks were occurring. It would have tools for them to calculate the quickest routes ambulances could take to reach those stricken by the deadly disease, or use that data to find the best places to set up temporary hospitals.

He found that this wasn’t as easy as it sounded.

“Lawyers, policy people, contracts officers all came into my office and they had lists of reasons why I can’t do that,” he said. “We pressed the envelope,” while ensuring that everything was legal, contractually authorized and followed policy, he said.

The resulting website had no passwords. It was a simple browser that allowed healthcare workers to quickly isolate those with the virus and shortened time between diagnosis and treatment. “That time saving led to life saving,” he said.

Things went smoother after the Nepal earthquake in April.

Within 24 hours, a response team at NGA’s headquarters in Springfield, Virginia, had set up an open website, posted atlases of major affected cities as a baseline and placed 240 data layers on top of 46 continuously updated maps for U.S forces, nongovernmental agencies and allies to consult. Digital Globe, a commercial satellite imagery provider, allowed its data to be used free for 30 days. Tens of thousands used the website to help them during rescue missions. This type of open collaboration needs to become the norm, Cardillo said.

“We did well in West Africa and Nepal, but we need to accelerate our progress and make content available and as accessible as possible and find better ways to leverage the contributions of others,” Cardillo said.

The latter part of that statement points to the other definition of “openness,” which is the ability for NGA analysts to exploit the growing amount of data coming from commercial and public sources.

“For decades, intelligence was like a regulated currency. We guarded it jealously. We controlled it tightly.” Hording data gathered through clandestine means is still useful, he said. “But in today’s world, our enterprise must operate differently. Less like a currency, and more like a current.”

Samuel J. Gordy, integrated systems group president at Leidos, one of NGA’s major contractors, said Cardillo is leading a “seismic shift” in the way the intelligence agency operates. He was a contemporary of the director when they both entered the geo-int field during the Cold War era.

“We were looking at Soviet missiles, ground-based missiles and tank armies and submarines and all that, and there wasn’t a whole lot to open source,” he said in an interview.

“The whole world has shifted. There is so much data out there. It is doubling, tripling, quadrupling over time. And it’s not a matter of generating the data. It is all being generated independent of us,” Gordy said.

The belief among the older generation that grew up in the Cold War was intelligence gathered from sources outside of the spy agencies wasn’t to be trusted or wasn’t as valuable.

“It’s not good data if it’s not classified data. That literally was the mind set,” he said. “Now, not only is that data out there, you have to be accessing it.”

The data glut is coming from a variety of sources, from commercial overhead imagery and live news feeds, to Twitter and other popular social media websites that stamp public posts with geospatial and time data, he said.

Cardillo at the conference released a new strategy document in order to crystallize where he thinks NGA should be going in this new world of open source data. He admitted at a press conference later that he is encountering institutional resistance. He likened it to a car with a standard shift. He is trying to step on the gas. Some are easing into the idea by engaging the clutch, while others are applying the brakes. “They don’t want to go there,” he said.

“We need more gas. That’s the mindset behind the strategy,” he added.

“These aren’t bad people and they’re not doing the wrong things. They’re just doing what they did five years and 10 years ago. This will be a continually educational process,” Cardillo added.

“I’m happy to challenge those policies, or take an alternative view that perhaps what got us here — and what got us here was a highly classified, highly controlled environment — isn’t going to take us forward,” he said.

The new strategy is about covering more ground and activity over time and threats of all kinds.

He wants to seamlessly weave traditional and nontraditional intelligence sources to present visually compelling geospatial-intelligence narratives.

It will be less about the analysis of an image and more about identifying the patterns across the images, across the spectrum and around the clock, he said.

Identifying objects from above will always be a part of what the agency does, but “our value proposition must become more than that. Our expertise will ride on top of the objects, infer insight, create understanding about significance,” he said.

A coming small satellite movement is going to “darken the skies” with commercial sensors. There are more than a dozen constellations being planned with hundreds of small spacecraft that will continuously scan the Earth, he said.

Some are uncomfortable with this, he said. “While I recognize that there are two sides to the world’s growing transparency, I’m energized and enthused about this development. ... Frankly, it has pushed geo-int to an inflection point,” he added.

“That [small-sat] revolution will do just that: revolutionize the way we sense the planet,” he said.

NGA will need help from vendors, academia and those outside the traditional military contractor community to help NGA make this transition, he said.

Yet there are still restrictions and barriers to accessing data, said Susan M. Gordon, NGA deputy director.

“We ought to expose our content in a way it can be used,” she said during a panel discussion at the conference.

There are examples of analysts who have to leave their desk and book time on computers — some of them in other facilities — to access the information they need. Cardillo lamented the fact that employees must still lock their personal smartphones up before entering the building. It is a potentially valuable tool going to waste.

“We simply have to get our data into the common stores so it can be used,” Gordon said, referring to cloud services.

And NGA needs applications that can be used to organize the new glut of data. It has established a new chief data officer position to help sort through the problems, Gordon said.

“If you don’t know where your data are, if you don’t what data you have, if you don’t know how to use it, if you don’t know what standards need to apply” then it is not being exploited to the fullest, she added.

Leidos’ Gordy said: “We’re okay with moving unclassified data up to classified levels. We are nowhere near where we need to be on moving classified level data down. … We’ve all grown up on protecting sources and methods.”

Hugh McFadden, manager of emerging intelligence programs at Northrop Grumman Corp., agreed that NGA is going to have to look beyond traditional military contractors for solutions to its big data problems.

“We can’t just look at the defense industry for solutions anymore especially because there is so much spatial analytics, spatial computing, spatial management, or content that is somehow adjacent to or valuable … that come from the commercial side,” he said in a press briefing.

Northrop Grumman is looking at partners outside the defense industry, either large companies or small startups that have done commercial work in data management. The underlying technology at these firms may apply to some of the tough problems NGA and other agencies are having correlating the ocean of information. It’s a matter of finding the right technology and adapting it to government use, he said.

Gordy said Leidos’ Digital Edge, an information technology platform widely used in the intelligence agencies, and increasingly in civilian departments, helps analysts sort through the data through queries. The hard part is ingesting all that data being gathered in a timely fashion, and taking it from different sources.

It can handle tens of billions of records a day, “and we’re probably not keeping up,” Gordy added. There are more data sources of a diverse nature, and correlating those is tough. “And the problems are becoming exponential because the amount of data just continues to multiply. We’re in a data overload situation and it’s just getting worse.”

Cardillo said he is looking for tools that will allow analysts to see beyond their niches. In briefings, he often says there are bigger stories that analysts are missing. There are political arcs, cultural arcs and human geography arcs. “You can see their heads start to spin,” he said.

A Syrian or Iranian analyst, or a hyperspectral analyst, is by definition a specialist and his or her understanding might be narrow.

If industry could help that analyst contextualize better by putting the larger picture on a desktop, it would let that expert know where they fit in that broader story, he said.

Cardillo added: “Our proud past has set us up for an amazing future. However, what got us here, won’t get us there.”

Topics: C4ISR, Intelligence