How Defense Agencies Can Better Cope With Big Data

The Defense Information Systems Agency’s five-year strategic plan says big data capabilities are becoming essential to modern warfare. Within defense and intelligence agencies, big data holds promise because of the strong potential to broaden and deepen their understanding of pressing issues.

Big data reporting, analysis, visualization, integration and development tools are vitally important for turning sizeable amounts of intelligence data into useful, collaborative information that can be efficiently distributed across military and intelligence communities.

Military leaders need to be able to access a comprehensive view of all of the data available, and the information needs to reach the hands of decision makers before it is old, if not obsolete.
National security agencies need to be able to integrate and deliver data in an agile and secure manner.

Because of the exploding volume of data assets and a growing number of fragmented data sources, it is an ongoing battle for defense and intelligence agencies to access integrated, relevant data for cross-functional use. New data virtualization technology is helping to overcome the challenges resulting from information existing on different platforms and in different environments.

Data virtualization integrates data assets from disparate sources, regardless of the location of the source, in order to create unified, virtual data views for any user or front-end solution such as applications, dashboards, portals, intranet or search.

Big data and data virtualization technologies are evolving rapidly. However, the adoption of data virtualization by defense and intelligence agencies remains a work in progress. Turning big data into actionable insights that can be rapidly disseminated to key users will require overcoming several hurdles that are currently hampering data virtualization from delivering truly transformative results.

One of the obstacles is the government procurement process. Traditionally, because of procurement rules, it takes a long time for government to react to new technology. Today, agencies are under increased pressure to modernize their applications and IT environments. This requires that agencies and contractors think differently. New agile approaches are needed in order to implement innovative technology and news ways to gain efficiencies and drive improvements to project quality and delivery without being labor or budget intensive.

Another issue is the “newness” factor. Agencies are generally risk averse. As a result, the introduction of any new, disruptive technology takes time for them to embrace. As with any new technology, it must go through the certification and accreditation process.

New technologies and approaches often require an internal change in culture. Quite often, there is considerable debate regarding who actually owns the data — application development teams or IT operations teams. The success of any new technology will necessitate the need to internally identify how the various teams work together to get the most out of the solution.

Big data requires a new way of thinking and a new approach to data management and analysis in order to avoid getting buried under the massive mounds of information that military organizations generate.

A significant benefit of virtualizing the data and applications is that it enables self-service data management and eliminates the amount of time it takes to make, refresh or reset copies. The amount of information being collected has overwhelmed analysis systems and processes. Unlocking data from its existing infrastructure and delivering it quickly and securely to the right location via data-as-a-service enables greater automation of intelligence and data.

For defense and intelligence organizations, data virtualization technologies would permit better information sharing, timely delivery and increased knowledge for faster intelligence insights.

Data virtualization ensures that data is well integrated with other systems so that organizations can harness big data for analytics and operations. Solving big data challenges requires the management of large volumes of highly distributed data stores along with data-intensive applications. U.S. intelligence agencies need to find ways to improve the integration of disparate pieces of data to coincide with their use of big data technologies, such as Hadoop.

By connecting the data dots in order to make the data smart and digestible for military leaders to actually use for better informed decisions in real time, virtualization provides the added level of efficiency to make big data platforms a reality.

Hadoop — an open source software — delivers enormous processing power with its ability to handle virtually limitless concurrent tasks and jobs across the cluster. Together, cloud computing and the Hadoop platform enable the distribution of large data sets within any environment and make big data easier to manage.

These technologies, along with analytical and visualization tools, will continue to have a major impact on military and intelligence agencies.  Future advantages afforded by big data are dependent on the ability to store, access and analyze unique data and deliver this information through different networks to the point of need, better and faster than our enemies. This is a big reason why data virtualization plays a critical role in producing actionable intelligence in a timely manner.

Virtualizing the data lets it live wherever it is already located, but provides integrated access to the users. This accelerates the time it takes to make data actionable and achieve better outcomes while removing concerns about how to access, store, convert, integrate, copy, move, secure and distribute the data upfront and on an ongoing basis. The end result: military leaders can access the data when and where they need it.

A cloud-based, agile approach to data management gives agencies the ability to simplify and automate the data migration process from legacy systems to the cloud, resulting in more successful transitions. This is accomplished by virtualizing the data and application stacks, enabling agencies to move exact copies of data and apps in minutes.

By utilizing data virtualization to make the management and analysis of big data more effective, defense and intelligence agencies can spend less time and money overseeing intensive processes and focus more on turning the data into action thereby improving military operations and mission success.

Ted Girard is vice president of Delphix Federal.

Topics: Infotech