Tackling the Cost Complexity of Observability Tools

Executive summary

As today’s modern businesses rely heavily on digital services to run day-to-day operations and drive revenue, the imperative is clear—those applications must function flawlessly. Customers have no tolerance for latency, bugs, or application outages—which, in turn, has propelled the significance of observability platforms. Engineers, tasked with building and managing applications, require a comprehensive view into the health and behavior of their applications and infrastructure. This insight is crucial for swiftly identifying and addressing issues, making observability platforms invaluable.

Notably, Gartner reports that 70% of organizations successfully leveraging observability will not only reduce latency but also gain a competitive advantage in their targeted business or IT processes. However, a pressing question echoes in the observability user market: How can one harness the power of observability data for a competitive edge without incurring exorbitant costs?

The palpable fiscal strain on organizational observability budgets extends beyond mere financial considerations. It's not solely about the cost of goods sold; a growing realization underscores that the value derived from observability tools is increasingly detached from their price.

This paper delves into the reasons behind the observability financial strain and explores how a paradigm shift to “observability 2.0” empowers organizations to course-correct their budgets. This modern approach to observability proves to be transformative, allowing organizations to achieve economic gains while delivering enhanced value.

Observability 1.0’s cost multiplier effect

To understand the current economic constraints associated with observability, a good starting point is to delve into its origins as a first-generation solution—"observability 1.0." The prevailing notion often attributed to observability 1.0 is that it has three pillars: metrics, logs, and traces.

From an evolutionary standpoint, the progression is evident. Initially, organizations had logs, prompting the creation of log aggregation services. The demand for dashboards led to the acquisition of metrics tools. As software engineers sought to instrument their applications, application performance monitoring (APM) tools were designed to help them understand real-world performance. Microservices then prompted the need to evolve from APM, and tracing was the answer, with its ability to help engineers understand requests to go across hundreds—or even thousands—of services. This trend continued with real user monitoring (RUM) tools addressing front-end engineering needs.

The consequence? Organizations find themselves paying to collect and store telemetry in multiple ways across various platforms, with a modest 5x multiplier being conservative. Many companies also bear the cost of multiple overlapping tools within the same category, such as profiling data, product analytics, business intelligence data, database monitoring/query profiling tools, mobile app telemetry, behavioral analytics, crash reporting, language-specific profiling data, stack traces, hosting provider metrics, and more.

Calculating an organization's unique cost multiplier hinges on the frequency of paying to store data about user requests. While various tools may collect slightly different data for specific use cases, the underlying data types—metrics, unstructured logs, and structured logs—remain constant. Each of these possesses distinctive tradeoffs in terms of cost and the value organizations can derive from them.

Metrics

Metrics, at their core, consist of a single number, often with attributes (for example, tags or labels) appended. But there's a catch—they discard all of the context of the request at write time. So, each number stands alone. This makes it impossible—from a cost perspective—to correlate one metric with another from the same request, select all the metrics for a certain request ID, user, or app ID, or ask any random questions about the metrics data.

Metrics (and logs, but we’ll go into logs later) are utilized in both RUM and APM tools. When implementing a metrics tool, it typically arrives preloaded with fundamental metrics. The challenge arises as the cost of custom metrics predominantly shapes your billing structure. Adding more custom metrics means the bill goes up at least linearly. To keep your expenses from growing uncontrollably, you need to frequently audit your metrics, try to predict which ones will be valuable in the future, and trim any you can do without. Even with experts at the helm, these tools demand substantial supervision.

Ideally, metrics costs would be predictable. However, the cost of each metric can fluctuate widely based on factors such as its composition, values, frequency of use, and other variables. Given the likelihood of numerous custom metrics per service, determining their individual costs becomes a challenging endeavor akin to solving a mystery. Metrics bills, whether intentionally or not, often remain veiled.

Relying solely on a metrics tool falls short when aiming for a comprehensive understanding of your software or system. Let's delve into logs and explore how they offer richer data, with consideration for their impact on observability costs.

Unstructured logs

Logs, the bedrock of understanding code intricacies, provide more insights than metrics. Logs are typically emitted multiple times throughout the execution of a request with one or a small number of nouns per log line, plus the request ID. Unstructured logs are still the default for application monitoring and observability, although this is slowly changing. This change is tied to the cost of unstructured logs, which is driven by a few things:

• Write amplification: If you want to capture lots of rich context about the request, you need to emit a lot of log lines. If you print out just 10 log lines per request, per service, and you have half a dozen services, that’s 60 log events for every request.
• Noisiness: It’s extremely easy to accidentally blow up your log footprint yet add no value—like placing a print statement inside a loop instead of outside the loop. Here, the usefulness of the data goes down as the bill increases.
• Constraints on physical resources: Due to write amplification, logging everything desired for all requests or users can be physically impossible. As a result, people often resort to blindly slash the log volume of log levels, consistent hashes, and dumb sample rates. When you emit multiple log lines per request, you duplicate a lot of raw data; sometimes over half the bits are consumed by request ID, process ID, timestamp. This can be quite meaningful from a cost perspective.

All of these factors present challenges, and a drawback of unstructured logs is their reliance on full-text search for querying. As log data volumes increase, the search speed decreases. For developers, this means searching historical logs over extended periods can take minutes to hours, which means experimenting and looking around for unknown-unknowns is prohibitively time-consuming. You have to know what to look for in order to find it. The result: as your logging bill goes up, the value goes down.

Structured logs

Structured logs are gaining prominence in the industry, particularly with the rising popularity of OpenTelemetry. What sets structured logs apart is their ability to go beyond sluggish, rudimentary string searches. Properly structured data enables not just efficient searches but also opens the door to things like calculations, percentiles, and heatmaps.

Tools built on structured logs are clearly the path forward. However, merely adding structure to your existing logs isn't the silver bullet. If you convert your existing log lines into key-value pairs, the challenges of amplification, noisiness, and physical constraints persist—though with improved search efficiency and data manipulation capabilities.

To make the most of structured logs, several measures can be taken:

• Canonical logs: Instrument your code following the principles of canonical logs, consolidating all vital request characteristics into a comprehensive, information-rich event. The value of this approach cannot be overstated, contributing to enhanced usefulness, usability, and cost control.
• Convert structured logs to traces: By adding unique identifiers to the logs emitted by each service in a request, you can turn them into traces, allowing visualization of end-to-end performance.
• Columnar storage engine: Direct your data into a columnar storage engine to eliminate the need for predefined schemas or indexes, providing flexibility for future searches and computations.
• High-cardinality support: Use a storage engine with high-cardinality support and an intuitive interface for exploration.

As you progress in enriching your structured logs, instrumenting your code with pertinent data, and presenting it in real-time, you unlock an entirely new realm of capabilities—an approach aptly described as "observability 2.0." To comprehend the nuances of this evolved 2.0 cost model, let's first examine how ballooning costs are embedded in observability 1.0.

Unraveling the cost tangle in observability 1.0

To summarize: the observability 1.0 model comes with inherent high costs. Each pillar of this model (i.e., metrics, logs, and traces) comes with its own price tag. You find yourself collecting and storing data repeatedly, incurring storage costs for each use case. Depending on your toolset, your observability bill may grow—and outpace—the growth of your traffic 3x, 5x, 10x, or more.

The situation worsens as costs rise, inversely impacting the value you derive from your tools:

• Log searches become increasingly sluggish.
• You must know precisely what you're searching for to find it.
• Blunt force sampling techniques are employed to prevent log volume explosions.
• Every time you want to ask a new question, you're obliged to commit new code and deploy it.
• Predicting which custom metrics and fields to index becomes a challenging guessing game.
• As volumes surge, your ability to uncover unknown-unknowns dwindles proportionally.

Moreover, there's no connection between these tools. Correlating a spike in your metrics dashboard with the corresponding requests in your logs, or tracing errors between different tools, is an impossible feat. If your APM and metrics tools report conflicting error rates, resolving this confusion is an uphill battle. The only connection between these tools relies on the intuition and guesswork of a few seasoned engineers.

Consequently, cognitive costs are substantial, and the risk to your team's bus factor is alarmingly real. The vital interlinking data in your system—connecting metrics with logs, and logs with traces—exists solely in the minds of a select few.

Simultaneously, the engineering effort required to manage these tools (and their bills) continues to grow. With metrics, engineers spend time auditing, fixing poorly constructed metrics, and removing those that are inefficient or unused. With logs, engineers monitor the volume, sift through duplicate log lines, and maintain indexes.

However, the most significant cost isn't the time spent grappling with observability 1.0 data types. The true expense lies in the unseen costs inflicted on your engineering organization—slowed development, mounting technical debt, and diminished confidence due to low visibility.

With these challenges in observability 1.0 firmly in mind, let's delve into observability 2.0 to uncover how it offers organizations a paradigm shift in addressing their observability needs and revitalizing their bottom line.

The shift to observability 2.0

Observability 2.0 marks a departure from observability 1.0’s traditional metrics, logs, and traces paradigm; it converges into a single source of truth. Anchored on wide, context-rich structured log events—known as spans—observability 2.0 transcends the limitations of its predecessor.

To understand how this approach fosters a more economically viable model, let's explore the main aspects of observability 2.0, which encompasses:

• Arbitrarily-wide structured events: Breaking free from the constraints of the three-pillar paradigm, observability 2.0 presents a singular source of truth. With this modern approach, you get infinite custom metrics since you can append as many as needed to the same events. This is an important touchstone of the 2.0 approach because, without this capability, you risk limiting the dimensionality and contextual richness of your data. Embracing highly-dimensional data with dozens to hundreds of dimensions per event provides contextual insights.
• High-cardinality data: So much of debugging is about finding needles in the haystack within the system, and high cardinality is what allows you to track down very fine-grained needles. For instance, querying for specific subsets like "Canadian users on iOS11 version 17.3.1 utilizing the French language pack who installed the app last Tuesday" becomes a streamlined process. Each of these parameters represents a high-cardinality dimension, crucial for granular analysis.
• Context-rich traces: Tracing the trajectory of a request through the system provides invaluable insights into patterns and where failure points are occurring across services. By preserving the contextual details, observability 2.0 equips you with the details you need to find the root cause of an issue and on how to best proceed with the fix.
• Enhanced explorability: By enabling seamless exploration of application performance, observability 2.0 helps foster a culture of curiosity within your team. Whether navigating through traces or breadcrumbs, you can adeptly explore different paths that can lead to new learning and awareness of app performance. With a unified data source, correlating disparate events becomes effortless, liberating analytical exploration from the constraints of cardinality and key space limitations.
• Distributed column store: At the heart of observability 2.0 lies a purpose-built distributed column store, uniquely constructed to accommodate wide events and examine high-cardinality data to provide rich context. This architecture prioritizes fast queries, eliminating the need for rigid schemas or pre-aggregated data. The absence of indexes streamlines data ingestion and query performance, ensuring you get expedited access to critical insights.

Shifting focus to finances, let's examine 2.0’s impact on the budget equation. The beauty of observability 2.0 lies in its ability to enrich telemetry without proportionally increasing costs. Instead, your telemetry becomes richer and more valuable as you add more key-value pairs. Additionally, the versatility of observability 2.0 extends beyond numerical data, accommodating a diverse range of information, including high-cardinality fields such as "App Id" or "Full Name."

Transitioning to observability 2.0 brings its own amplification factor into the equation. As you enhance your code with additional telemetry per request, the number of events to send (and pay for) naturally rises.

That said, with observability 2.0, you have access to robust tools like dynamic head-based sampling and tail-based sampling that grant you control over whether to retain a trace after its completion, thus seamlessly managing control over increased event volumes. This capability ensures you can capture 100% of slow requests and other outliers with precision, with the added benefit of minimizing your observability costs by discarding requests that you don’t need.

This economical approach of observability 2.0 marks a fundamental departure from observability 1.0.

Recouping engineering time: The currency of innovation

Another significant difference between observability 1.0 and 2.0 isn't merely confined to substantial budget savings—it manifests in the engineering team's ability to save time and easily navigate complexities. This transformation empowers them to gain profound insights into the code, as well as explore, iterate, and deliver with confidence.

Modern software engineering is all about hooking up fast feedback loops. And observability 2.0 tooling is what unlocks the kind of fine-grained, exploratory experience the team needs in order to accelerate those feedback loops.

Where observability 1.0 is about mean time to respond (MTTR), mean time to detect (MTTD), reliability, and operating software, observability 2.0 is what underpins the entire software development lifecycle, setting the bar for how quickly development teams can build and ship software, find problems, and iterate on them. Observability 2.0 is about being in conversation with the code, understanding each user’s experience, and building the right things.

Conclusion

In the observability evolution, the shift to 2.0 isn't just a leap—it's a strategic advancement in sync with the demands of modern software engineering. Embracing observability 2.0 is an investment in efficiency, adaptability, and a future-ready engineering plan. With an eye on cost savings and streamlined efficiency, this shift promises to redefine an organization’s observability landscape.

Navigating the many priorities within engineering is certainly no small feat. To help assess the value a move to observability 2.0 would provide your organization and determine where to prioritize it in your roadmap, consider these guiding questions:

• How effectively do current observability tools align with development speed and confidence?
• Are there noticeable hindrances in obtaining meaningful insights and managing current debugging practices?
• What are the potential economic gains and enhanced value that observability 2.0 could bring to your organization?
• How could a shift to observability 2.0 impact the company’s overall software development lifecycle?

If you need to discuss these questions further, we’re happy to help. Book time with one of our experts today.