How 2024 prepared us for a new era in observability

As we reflect on the past year, one thing is clear: 2024 marked a tipping point for observability. Driven by the exponential costs of downtime — a staggering $400 billion annually for Global 2,000 companies — and the complexities of modern cloud ecosystems, it has evolved from a niche consideration to a strategic imperative! 

The benefits have been undeniable – leaders today resolve issues 2.8 times faster, reduce downtime significantly, and deploy code on demand with greater success. Yet, this rapid adoption has come at a price — literally! The startling revelation by Honeycomb's Charity Majors, who said that observability costs can devour up to 30% of infrastructure spending, has left many reeling. 

Fortunately, the industry is poised for a correction. Breakthroughs in data management, vendor-agnostic architectures, and AI-driven intelligence are converging to democratise observability. OpenTelemetry and GenAI are unlocking new efficiencies, empowering companies to slash costs, reclaim data control, and anticipate problems before they arise.

Now, let's dive into the specifics of this shift and explore how they are making observability tools more accessible, affordable, and effective for enterprises, SMBs, and startups alike.

Data Control Done Right!

In recent years, the shift to distributed microservices architecture, mass data collection, and complex pricing models have significantly driven up observability costs. This perfect storm has left organisations struggling to manage their expenses.

Replacing single host servers with dozens of microservices generates exponentially more data, which directly impacts observability bills. Compounding this issue, observability providers' agents collect data indiscriminately, without distinguishing between essential and non-essential information. This lack of discernment leads to unnecessary expense.

Legacy observability companies have also instilled fear into organisations, leading them to believe their systems could be compromised if they don’t share all their data. As a result, organisations are sharing more data than necessary, further driving up costs. Since observability cost is tied to the volume of data, which varies every month, organisations can’t accurately estimate their expenses.

Believe it or not, nearly 70% of collected data is unnecessary! Yet companies continue to pay for it. This unnecessary data collection presents a significant opportunity for savings. By reducing unnecessary data sharing, companies can cut observability costs by 60-80% without compromising performance or insights. 

Observability Has To Be Flexible

Traditionally, observability companies have locked in customers with costly and cumbersome contracts, making it difficult for businesses to switch providers. Financial constraints and fears of business disruptions have held companies’ hostage.

However, the tide is turning. OpenTelemetry has standardised APIs across platforms, leveling the playing field. This innovation enables companies to seamlessly migrate systems and data between vendors without incurring excessive costs or technical complexities.

While OpenTelemetry is a significant step forward, implementation and integration challenges remain. Some vendors are also developing proprietary extensions or integrations on top of the framework.

Newer observability players are also offering flexible, pay-as-you-go models. This approach empowers companies to choose between long-term commitments or shorter-term arrangements, eliminating the fear of vendor lock-in.

AI's True Potential in Observability

Artificial intelligence is neither a silver bullet nor snake oil! Though AI has the power to take things up a notch, its promise is often overshadowed by exaggerated claims and misconceptions. 

One pressing pain point for our customers is the time-consuming process of debugging issues. Sifting through vast amounts of data to identify root causes can be costly and detrimental to business operations. AI can automate and streamline this process, saving developers nearly half of their time. For instance, AI-powered observability can predict system crashes based on CPU memory usage, providing proactive alerts to customers.  The integration of Generative AI (GenAI) into observability further simplifies the complexities of accessing critical insights. 

It’s great to see that the industry is making strides in this direction, and the results are promising! 

This is Just the Beginning

With 87% of organisations already leveraging observability tools, the divide between leaders and laggards is stark. Those who have achieved full-stack observability reap remarkable benefits – 79% less downtime and 48% lower outage costs. But here's the exciting part:  As barriers to entry continue to dissolve, observability is poised to become even more accessible, affordable, and widespread. 

However, merely onboarding an observability provider is just the first step. To unlock true value, organisations must prioritise vendors that are adaptable, agile, and committed to delivering tangible return of investment (RoI). It's time to look beyond the hype! Instead focus on partners who can evolve with your needs, customise solutions, and drive lasting impact.

The observability journey demands more than just technology – it requires a forward-thinking mindset. By choosing the right vendor and embracing a culture of continuous improvement, organisations can unlock the full potential of observability and propel their business forward.

Laduram Vishnoi

---

Laduram Vishnoi, Founder and CEO at Middleware.io (YC W23).