Data Cleansing Improves Federal Government Outcomes

iStock illustration

The 2024 fiscal year budget released by the Defense Department has $145 billion earmarked for research, development, test and evaluation funding, including $1.8 billion for artificial intelligence and $1.4 billion for systems like joint all-domain command and control, also known as JADC2.

The budget clearly prioritizes the continued investment in modernizing and innovating the department’s operations.

Investments of this scale in AI, machine learning, data analytics and other emerging technologies can be undermined by poor data hygiene.

Even when info-tech systems are capable of processing enormous amounts of data, the adage of “garbage in, garbage out” still applies. Without accurate, reliable and sortable data, systems like the Federal Procurement Data System will only provide limited insight and diminished return on investment. Data quality cannot be achieved without strategic data cleansing.

Data cleansing — the process of identifying and correcting or removing inaccurate, incomplete or irrelevant data from a dataset — is not a one-size-fits-all approach. Depending on the different types of data being collected across various channels, agencies will need to determine what kind of clean-up is required to fully take advantage of advanced analytics.

With messy data, agencies risk relying on incorrect analyses that can lead to poor decision-making. By cleansing data, agencies can ensure that they are working with accurate, complete and relevant information that will actually improve operational efficiency.

With a cleansed data set, agencies can supercharge their IT system’s capabilities in several ways, including increased accuracy and efficiency. Data cleansing can help agencies save time and resources by automating the process of identifying and correcting errors, reducing the need for manual data entry and validation.

It can also facilitate better data integration. Data cleansing can help agencies integrate data from multiple sources more effectively, ensuring that they are working with a complete and accurate dataset.

It can also improve analysis quality. By investing in data cleansing, agencies can ensure that they are working with high-quality data that is reliable and trustworthy, which can help them build better models and make better decisions.

The ramifications of poor data hygiene for agencies can manifest in several areas, including audits. Consider the Defense Department’s financial audit process, for example.

The department has faced significant challenges in consolidating its financial management systems. Migrating billions of transactional records and associated dormant balances from legacy systems to a new platform is a daunting task that requires rigorous data cleansing. If inaccurate and unreliable financial data is propagated during the migration, the department risks compounding extant problems by continuing to use data in its previous form.

With proper data cleansing, including the implementation of automation solutions, the department was able to achieve rapid and powerful results, including producing auditable financial statements, executing effective and transparent budgets and complying with financial laws, regulations and policies.

Failing an audit is one negative consequence of poor data management, but the stakes can often be much higher. Uncleansed data being analyzed from multiple sources to identify patterns and trends in military activity may contain errors or inconsistencies, such as duplicate records, misspelled names or missing information.

Consider the possibility that an AI-powered platform is fed unclean data, such as inaccurate satellite images, incomplete information on enemy units, outdated troop locations and inconsistent threat classifications. As a result, the AI system may make incorrect predictions about enemy force strategies that result in the military misallocating resources and personnel, wasting resources and time at best or causing undue casualties at worst.

Defense decisions are ultimately made by humans, but these decisions are only as intelligent as the information supplied to military leaders. If leaders are unaware that the projections and assumptions provided by their tools rely on unclean or incomplete data, they may inadvertently make choices that compromise the nation’s defense posture.

The challenge of system migration is substantial, but the Department of the Navy provides an instructive example of how data cleansing can reduce this kind of burden.

It needed to migrate billions of financial records, including unwieldy aged or dormant unliquidated orders, unmatched transactions and abnormal accounts payable transactions, from old systems to the Navy Enterprise Resource Planning platform. To make sure the new system worked properly, the Navy needed to pare down the sheer amount of data to a more useful and effective data set, validate accounting information and fix errors through data cleansing.

By embarking on the process of data cleansing before each migration step, the Navy is actively simplifying the process and will require fewer resources once data is moved. Accurate financial data helps it produce auditable financial statements and budgets, not just abiding by legal requirements, but building a knowledge base that can inform better decisions.

Committing to strong data cleansing efforts at an early stage is crucial in boosting the accuracy and effectiveness of the Navy’s financial management processes and ensuring responsible stewardship of taxpayer dollars.

Investing in data cleansing solutions early in an agency’s digital transformation journey won’t just save agencies time in the long run but will bolster the level of insights agencies can draw from their collected data. This approach leads to auditable financial statements, transparent budgets and compliance with financial laws.

To make the most of the advanced tools at their disposal, government agencies must prioritize data cleansing when developing IT systems designed to achieve government transparency, efficiency and compliance. ND

Patrick Sweeney is the vice president for the Financial Systems Integration Practice for Aeyon, where he focuses on the strategic direction and growth of the firm’s financial systems community of practice. He has 36 years of Defense Department financial management experience.

Topics: Budget, Defense Department