The Data Observability advantage in business decision-making

Organizations are increasingly trusting data to augment their decisions. However, data is not always of consistent quality or desired accuracy. In a report published in November 2022, market research firm Gartner, said, the average cost incurred by organizations annually due to poor data quality amounts to $12.9 million. The problem is compounded due to a large number of data sources. 

As per industry estimates, most organizations deal with more than 400 sources of data. Organizations need trustworthy data to power up decisions across functions and operations with a lasting impact on the customer experience and revenues. This builds the case for data observability.

Data observability refers to a vast range of activities and technologies that help understand the state of data in the system. It allows data professionals to assess, pinpoint, troubleshoot, and resolve data anomalies such as pipeline failures, errors, and poor quality in near real-time. It helps improves their visibility into data systems and understanding the data health, thereby, enabling faster, automated problem identification and resolution.

Traditionally, organizations have relied on application performance monitoring (APM), and network performance management (NPM) systems. But with technological development climbing new heights, nearly every organization aspires to be a tech-first company. Agile development practices, continuous integration and deployment, DevOps, multiple programming languages as well as a host of cloud-native technologies are becoming more commonplace. 

These are essential to an organization’s strategy of ‘innovate and get to market briskly,’ which means new application components must be ready to be deployed rapidly in different places, in multiple languages and within a fraction of a second.

Here, data observability offers a more sophisticated solution towards addressing the highly complex, modern cloud-native application and data environments. It is a natural evolution of traditional data monitoring systems and uses historical trends to ascertain data workloads and pipelines directly at the source, assess their performance, and identify problem areas if any.

Observability platforms work by gathering telemetry in various forms. Telemetry is the measurement and wireless transmission of data from remote sources to the IT system for analysis. Records of each event thus provide developers with a ‘play back’ mechanism when they want to debug or fix data issues. The collected data is then correlated and shared continuously with associated teams – DevOps, Site Reliability Engineering, IT etc., equipping them with complete and comprehensive information. This keeps them aware of performance issues in an application or system in real-time.

Let’s say, it usually takes a user less than two minutes to file a query on an organization’s website. However, for some reason, if it takes longer, the IT team will need to trace back to the code to identify the issue. Observability provides the kind of end-to-end transaction level monitoring required where it becomes possible to trace the problem down to the last level of code.  It offers IT teams, specifically data engineers, the accuracy of data sets, and helps them to understand data lineage better.

In this competitive landscape, organizations are constantly looking to improve their speed of response, decision-making and optimize business operations. The overarching benefit of observability is that with all other things being equal, a more observable system is easier to understand (in general and in great detail), easier to monitor, easier and safer to update with new code, and easier to repair than a less observable system. In fact, Gartner predicts that 70% of organizations successfully applying observability are likely to have shorter latency for decision-making by 2026.

To summarize, data observability will help bring robustness and efficiency to data platforms and associated pipelines. As data complexity grows in terms of variety, volume and veracity, data observability will the key enabler for the organization to manage, monitor and extract trustworthy insights from the data. Hence, it is now imperative that business leaders along with technology stakeholders evolve an enterprise-wide data observability strategy with the help of correct telemetry, storage, analytics and altering tools.

Sameep Mehta

---

Sameep Mehta is a Distinguished Engineer at IBM Research India.