Everything You Ever Wanted to Know About Distributed Tracing (But Were Afraid to Ask)

“The map is not the territory, the word is not the thing it describes.”
— Alfred Korzybski

We live in a world of staggering systemic complexity. Eight billion human beings exist on this tiny blue dot, and the overall standard of living for those individuals is as high as it’s ever been. We have eradicated entire classes of disease, we grow plentiful food, and we span the globe with telecommunications that blanket the surface.

These systems do not come freely, nor do they come for cheap. The maintenance of these systems requires civilization-spanning effort through many disconnected, and disaggregated, groups of people. They must be resilient to disaster, subterfuge, and shock. Somehow, we push on into the darkness with a light at our side, an eternal quest to create brighter futures for the next generation.

That said, let’s talk about distributed tracing.

What is distributed tracing, and why should I care?

Distributed tracing, in short, is a way to output a normalized set of structured logs from applications which include contextual information allowing developers to view an end-to-end request. It’s a bit easier to discuss alongside the historical context of software systems since the late 1990s.

The early days: Dapper to Jaeger

Distributed computing has been a part of software for many decades, certainly before the advent of a company like Google. However, Google’s rapid rise to global popularity and its emphasis on performance led to the advent of new tools and techniques for performance monitoring. One of these systems was known as Dapper, which defined a large-scale distributed tracing system in production. The basic idea behind Dapper was simple: pass a unique context identifier through requests, and have each request independently write a log message out that contained this information, as well as the context information of the sender. These messages were known as spans, as each recorded the duration of work being performed in a service.

Dapper itself was not developed in a vacuum. Academic prototypes for systems like Pinpoint, Magpie, and X-Trace all provided the foundational work for Dapper to rise. Indeed, covalent innovation was happening at other organizations building planet-scale infrastructure. Zipkin, from Twitter, and Jaeger, from Uber, both helped popularize tracing in the broader software ecosystem by being open source and easy to use.

What ultimately characterizes this early stage of distributed tracing is its focus on visualization and analysis of an end-to-end transaction. For instance, developers could now track a single request across hundreds or thousands of microservices in order to determine which service contributed the most latency to a single request. Apache Hadoop developed a distributed tracing tool known as HTrace in order to understand the performance of MapReduce jobs.

Application performance management tools were often viewed as the commercial predecessor of open source distributed tracing solutions. Conceptually, they were similar: an APM tool allows developers to view end-to-end latency for a transaction by API route, database query, and so forth. Distributed tracing allowed these tools to broaden their remit and capture requests as they moved between services, adding more depth to their offerings.

By the mid 2010s, though, cracks in the promise of distributed tracing were starting to be seen—and they all had to do with interoperability.

Standardizing distributed tracing

A key insight for distributed tracing is that it requires distributed context. This means that services participating in a request need a single, pre-shared, and well-known set of headers and header values by which a unique identifier for a request can be communicated. Commercial APM solutions avoided this issue by promulgating their own proprietary specifications and formats for this data. Open source solutions each defined their own set of headers. This led to curious scenarios where a single request might have three or more tracers attached to it, each with their own tracing headers, duplicating work to produce what is fundamentally the same data.

In addition, the status quo was challenging for library and framework authors to participate in. While even commercial tracing tools had open APIs, the incompatibility between them meant that there was no ‘single answer’ for a developer to use in order to expose tracing to their users. The complexity of tracing support only increased as new technologies such as Kubernetes increased in popularity as well. In a few short years, many applications scaled up and out in dramatic fashion as K8s eased the adoption of cloud-native development and deployment strategies. Interoperability between container orchestrators, infrastructure, and application code in terms of tracing was unlikely to happen without standard approaches to creating traces, and providing for distributed context.

In the late 2010s, a project called OpenTracing was released in order to bridge this gap. OpenTracing defined a standard API for creating traces, and was meant for library and framework developers to integrate in order to have a ‘single answer’ to the ‘what should I use?’ question. Around the same time, Google released an open source project known as OpenCensus that tried to answer similar questions while also extending it a bit—rather than just having a standard API, what if there was a standard toolchain as well? What if the entire telemetry pipeline could be created and managed with zero proprietary code or agents?

The competition between these two projects eventually led to consolidation with the advent of OpenTelemetry and the introduction of W3C TraceContext. OpenTelemetry is designed as a trace-first observability framework, and W3C TraceContext is a single, globally supported format for creating and managing distributed context. Combined, these two projects opened the proverbial floodgates for distributed tracing as a core observability signal.

Tracing in 2025

We’re six years into OpenTelemetry at this point, and a few things are readily apparent. First, distributed tracing has become an integral part of the observability strategy at cloud-native organizations across the globe. No longer a ‘niche,’ OpenTelemetry and the ecosystem of tooling around it has turned tracing from a ‘nice to have’ to a ‘must have’ for developers and operators alike. Second, many cutting-edge features from the mid 2010s have been commoditized. Tail sampling, aggregate trace analysis, and span-to-metric processing have become far more commonplace in both open source and commercial products. Third, tracing is moving deeper into the stack. Rather than sitting at the application layer, tracing is being shipped as part of infrastructure tools (such as Kubernetes) or being run at the network layer via eBPF.

These gains are not equally realized, though, in the user community:

• Many developers still work in a world where traces are subject to significant random head sampling.
• One of the most common patterns for working with traces remains using a trace waterfall to visualize an end-to-end request.
• While instrumentation has become more commonplace and easy to use, developers still struggle with understanding the model and migrating away from traditional unstructured logging approaches to application observability.
• High data volumes and complex processing pipelines can lead to challenges in value realization.

In the rest of this whitepaper, I’d like to discuss these challenges. I believe that tracing isn’t just ‘nice to have,’ it’s the foundational signal for cloud-native observability. As per Ranbir Chawla of Ritchie Bros.:

“Getting to traces was a multi-complex undertaking, and reaching that stage was a priority for us. I was truly impressed by Honeycomb’s commitment to scaling that mountain alongside us to support our project of going from a logs-centric to a trace-first approach, and the results have been remarkable.”
— Ranbir Chawla, Senior Vice President of Engineering, Ritchie Bros.

Realizing these gains requires us to understand how and why tracing works in OpenTelemetry.

Why tracing is fundamental to OpenTelemetry

Think back to how this whitepaper started: a discussion of systems.

An immense amount of labor goes into the calculation of statistical data about the systems that power our modern society—everything from the volume of wastewater processed and what it contains to the total acreage of soybeans in Kansas. These statistics are used in myriad ways and influence many other systems. The price of a pound of soybeans is influenced by an unimaginable amount of factors, including historical trends in the weather and climate, geopolitical factors such as taxes and tariffs, transportation costs, and much more. We can, broadly, refer to these covalent factors as part of the context. Context is a crucial aspect for understanding systems. Without context, data points are isolated and uninterpretable. Context joins together like points and ensures that our analysis can be comprehensive—not simply containing what’s important, but also leaving out what is unimportant.

Similarly, OpenTelemetry understands that context is key for understanding a distributed software system. Thus, it builds on context as a ‘first-class’ citizen. It’s important to note that context, in OpenTelemetry, is quite literally separate from other components—it’s not part of tracing or logs. It’s shared between all of them. That’s because OpenTelemetry does something different from previous tracing systems—it rejects the concept of ‘three pillars’ (independent signals with independent analysis flows) and instead treats telemetry as a unified ‘braid’ of highly correlated context-rich data.

First-class context means that you can make some very different decisions about your telemetry data. Foremost, it should change the way you think about your telemetry data strategy. An overwhelming majority of cloud-native applications operate using traditional request-response semantics. This means that most interesting things about a service will happen as part of the request context. Traditionally, developers have relied on log frameworks to record the state of these transactions. In high QPS services, these logs are often paired with metrics in order to avoid expensive or complex log processing pipelines.

OpenTelemetry’s trace-first approach changes this fundamental calculus. Rather than relying on inconsistent logging behaviors, or low-cardinality metrics that can’t capture crucial transaction details, you can use spans as a fundamental log of each request handler or service in your system. A span contains the critical details needed for alerting—if a transaction succeeded or failed, how long a given request took to handle—while offering an easy way to add business context through free-form, schema-driven attributes. While it’s impractical to track the journey of a single tomato from seed, to vine, to plate, we’re able to accomplish such things in software.

How are traces made?

Let’s get practical for a moment. How does all of this work? It all starts with distributed context. Conventionally, OpenTelemetry relies on the W3C TraceContext standard to define how context is communicated. Each unique request generates a globally unique trace identifier when it starts. This identifier is stored in a context store, such as the request object, a thread-local variable, or some other in-memory mechanism. This identifier is incorporated into spans, which are units of work in a request.

Spans can be big or small—they can correspond to the work of an entire microservice, or a single function. We’ll discuss this more later, so keep it in mind. Spans have names, which you use to help identify them not only as a part of a trace, but for aggregation. For example, your service might handle requests at /api/v1/getScores. The name of a span for this route would be ‘/api/v1/getScores’. In addition to a name and duration, spans receive attributes that describe what the span represents. Our getScores handler would receive semantic attributes for things like HTTP Method, status code, and body size. You can add additional metadata here as well—conventionally, this includes business-specific data like customerId.

A few things to keep in mind about spans:

• Spans are immutable. You can’t modify them while they’re being written, you can only replace things.
• Spans will record until they’re stopped, usually at the end of a request.
• Spans can be used to record both synchronous and asynchronous work, but this does influence how you use them.

More on all of those points later.

As spans complete, they’re exported through the OpenTelemetry SDK to a configured destination, such as an OpenTelemetry Collector, or other compatible endpoint. There are a few things that can happen here, which I’ll list out below:

• Traces can be buffered and sampled in order to reduce noise in your overall dataset; for instance, keeping traces where errors happened, or preserving ones where latency was greater than some preset value.
• Metrics can be derived from span data, giving you an efficient way of creating measurements of ‘golden signals.’ This can be done at write-time or read-time, either creating these metric streams as traces are generated, or storing the trace data in order to create these time series when queried.
• Spans can be filtered or transformed in order to remove PII or ensure they conform to a schema using tools like the Transform processor.

Let’s walk through a few practical examples.

N-tier web application

If you’re working with a conventional web application (requests come from a web client, are routed through load balancers or gateways into a suite of service handlers, and finally to a database and back), then tracing is a natural fit.

• Either at your web client or at the entrypoint to your backend, you’ll start a trace. You can choose to treat these as independent traces as well, linked through span links (another OpenTelemetry concept) if you wish to keep a separation between the client-side telemetry and the server-side telemetry.
• Your ‘root span’ should be the primary thing you care about measuring from a performance perspective. It should be reflective of end-user experience with your application. If you’re using a gateway service like Envoy, it’s capable of creating and starting traces for you.
• When a span is created, the span context is created and stored in your service context store. This will vary depending on your language, and framework. For the purposes of this illustration, let’s assume you’re using HTTP. The span context will be stored in the request object.
• As work occurs in your service, you’ll add additional details to the span. Conventionally, a set of semantic (standard) attributes will be applied by the instrumentation for your RPC framework—things like the route name, or status code. The details that you add should be things that aren’t necessarily captured by these standard attributes, such as customer identifiers.
• When your service makes a remote call (either to another service or library) these calls will result in child spans being created. The span context is passed to the next service through some context carrier, such as HTTP headers, where it is extracted and used to create a new span. An important thing to note is that each span is independent. So, the parent span can’t change its name based on something that happens to a child, for instance.
• When the work performed by your service ends, the span ends as well. Attributes such as HTTP Response Code are set at this point, and the span is sent through a pipeline for additional processing, then exported.

These are ideal scenarios for tracing, to be sure. As you can see, the entire request fits neatly into a single waterfall. It’s possible to aggregate all requests to an endpoint with a given name, then determine what characteristics those requests that showed poor performance have in common. Honeycomb, for example, excels at these sort of aggregations over production data, allowing you to alert on not just anomalous traffic or high latency, but to use high-cardinality values as part of those alerts (for instance, complex matches between region and customer ID in order to properly route alerts to the right team, or to monitor VIP customer experience). Since the data is already neatly organized by API route, you can also use this to derive metrics for long-term storage, assuming that the QPS of your system is in the thousands.

Pub/Sub message queue

In this model, work happens asynchronously. A manager service may spawn tens or hundreds of individual jobs by publishing events to a stream processor like Kafka. Other services listen for events that they can handle, and when they see a new one, pick up the job and do some work before publishing the result back to the stream.

• Similar to before, the first service participating in the trace starts it. However, it’s not as simple as saying that there is a single ‘root span’ for your transaction. After all, it is possible that the manager has nothing to do with a job after it’s been dispatched.
• The publisher takes its span context and again injects it into a context carrier. In Kafka, this is the message envelope. Other stream processors may use different semantics. Regardless, the span context must be propagated to consumers.
• As consumers pick up jobs, they start their own spans. A few things differ here. First, consumers can elect to consume the context as a link rather than as a parent. A link has the side effect of consumers each creating their own distinct trace, rather than being a part of the parent trace. Second, and regardless of how the relationship between the spans is encoded, the span kind is set appropriately to Consumer. This metadata allows analysis systems to understand the different relationships (e.g., is the parent span blocking while waiting for the work in this child span to complete or not).
• Spans are created and exported much like our earlier example—nothing changes here.

This is the same simple producer/consumer, but the first uses parent/child relations and the second uses span links. The important distinction between the two is that the first one shows that the root span/producer spans stop before the consumer spans begin, and that there is some duration between the end of the producer write and the start of the consumer read. The other distinction is that in the second figure, all of the spans are in different ‘windows’ with dotted lines connecting the consumer roots back to the producer.

Tracing is extremely valuable to understanding Pub/Sub applications, although it requires a more sophisticated analytical approach. The most approachable analysis is a visual one, where you simply look at a transaction and see which consumers took the longest. You can alert on various exceptions, or even search for traces where a particular consumer took longer than some par time. Where it gets challenging, and more suited for post-hoc analysis, is when asking questions like “How long did the entire stream take to run?” or “Alert me when a given job has more than xx failures.” These can be challenging real-time questions to answer due to the complexity of the queries involved and the fact that there’s no way for the analysis tool to know when a given job is completed without some manual work (for instance, adding ‘stop’ attributes to the terminal operation in a job). You get most of the other advantages of tracing, though, such as per-consumer statistics, and an easy way to understand a specific end-to-end run of a job.

These two patterns cover a significant amount of software in production today. They’re general guidelines, as well. You may find that your needs require alternate implementations or strategies. Some async systems, for instance, may have multi-day waits between job steps. Some request/response systems may have security boundaries or data governance requirements that result in an incomplete view of system state. Tracing, while powerful, is not a panacea. For it to be truly useful, you need to combine it with other signals through unified telemetry.

Unified telemetry with OpenTelemetry

To understand a complex system, you must accept that there is an inherent complexity in the telemetry that system emits. Consider our example of food production: while you could understand the growth and consumption of a particular fruit, like tomatoes, by recording every single tomato that was picked along with all of the context around it (who picked it, where, the provenance of the seed, the fertilizers and chemicals used on the field, etc.) then transforming that data into the correct format at query-time, this is highly inefficient. The inefficiency can be derived from two rationalizations:

• First, most things are boring.
• Second, there are always known knowns.

Let’s walk through each of them in turn.

Any system is going to consist of an uncountable number of logical events. Whether we’re talking about tomatoes or B2B SaaS, granular analysis of a system will result in incalculable complexity. What’s the smallest unit of work when picking a tomato? Is it the actual act of picking—a hand grasping the fruit? Is it the motion of each finger as it squeezes the skin? While it’s possible to record actions exhaustively, most of this nuance and detail is unimportant in an aggregate analysis. Similarly, in a SaaS application, it is probably not useful to record every instruction that a CPU handles, nor the result of every function call.

There’s a further realization here, that in any system, there are only three possible states:

• Everything is fine.
• Everything is broken.
• And finally, ‘normal.’

In the outlier cases, fine-grained detail is unnecessary because it potentially obscures details we care about. It’s also extremely rare to have a system where everything is at one of these extremes. There are always errors and blips and one-off issues that are being handled by various parts of the system that have no appreciable impact on the performance of your application. This leaves us with ‘normal,’ an uncertain state. Some things are working, some aren’t. Your goal is to collect enough data, at the right resolution, to find the things that are suboptimal and repair them before they cause cascading failures.

This leads to known knowns. The primary benefit of tracing is that it gives you a lot of insight into the unknowns of your system, but not every problem is a novel problem. There are certain statistics that can either only be represented through other signals (such as disjoint resource utilization, like CPU, memory, or disk space) or are common enough that you should always care about them. Remember the golden signals? They’re golden because they’re always applicable, always useful, and easy to collect. They are known factors that are a useful indicator of system state.

OpenTelemetry’s approach to signals allows for the unification of telemetry in a way that allows you to address these inefficiencies. The API allows you to define telemetry with intent, and that intention should be respected by telemetry consumers. For example, our n-tier web application. While you can derive the golden signals from trace data, this does start to break down at high QPS. Creating, processing, and exporting spans requires more memory allocations, more CPU cycles, and potentially more network bandwidth than creating a single histogram at service startup, then pushing small changes to it over the network every minute.

This is why OpenTelemetry context is critical—because it can link datapoints together in a durable fashion, allowing for deeper analysis. Rather than independently recording spans and metrics, the metrics can be aware of the existence of spans. You can record a histogram that also writes an ‘exemplar’ (the associated span for a given metric datapoint) to the metric record. This data can be used by telemetry consumers in order to make better sampling decisions about what data to keep and what to discard, or for creating alerts that provide more detail and contextual information.

Rather than just seeing that your tail latency on a given API route is high, you can also see the specific requests that caused that spike. This also allows for greater flexibility in where the data is stored. For instance, if you can guarantee that potentially interesting traces are going to have a sparse representation at your alert layer (e.g., through the use of exemplars), you don’t have to pay the price of transmitting the entire trace until it’s needed. Those traces can stay closer to your workload, only flushed to the analysis tool when required.

This same principle applies to all other telemetry generated by OpenTelemetry. Log events can be stored locally with a reference to their parent spans, or filtered based on some other piece of information in a trace. Profiling data can be automatically recorded based on certain attributes from another service in a system. An entire world of routing, parsing, sampling, and dynamic modification is possible through the distributed context that OpenTelemetry provides.

The future of tracing—and how to be ready for it

As an industry, we’ve only started scratching the surface of what distributed tracing and OpenTelemetry can do. To wrap up, I’d like to offer a few thoughts on where things are today around tracing, where they’re going, and how you can be ready for the changes.

Sampling

Sampling is a method to control the amount of tracing data that you produce and ingest. Today, users typically practice two forms: head-based or tail-based. Head-based sampling makes a sampling decision when a request begins. Tail-based sampling makes a sampling decision when a request ends. Many users, especially those operating large-scale systems, use a hybrid approach—head sampling to reduce the overall volume of traces, then tail sampling to preserve only the most valuable.

Unified telemetry is going to dramatically impact sampling systems and strategies. Being able to rely on exemplars, for instance, improves the ability of sampling systems to capture the most valuable data. Additionally, these sampling techniques can be used as part of a reactive strategy that not only persists the right data, but also feeds back into an OpenTelemetry deployment in order to enable more detailed instrumentation as needed. In the short term, we’ll see more (and better) data management tooling developed around storage for unsampled data. Rather than dropping undesired spans, they will be written to cheap short-term storage and hydrated on demand for analysis.

Benefiting from these improvements requires two things. First, high-quality tracing instrumentation. OpenTelemetry gives you a lot of this out of the box. Second, integrating your application metrics with your OpenTelemetry spans. This may require more effort, as you will need to update your existing metrics API with the OpenTelemetry equivalents.

Asynchronous workflow tracing

In our earlier examples, we discussed a Pub/Sub system. Users often find that their biggest challenges with tracing involve similar async work with systems that have jobs which may run for days or weeks. Since spans are immutable and stateful while data is being recorded, this often results in traces with missing root spans, and an inability to visualize or alert on an entire end-to-end workflow.

There are active discussions around solving this issue in the OpenTelemetry project, some of which are doable today (with a bit of DIY smarts). For instance, you can write a custom processor that emits log events when spans start, end, and periodically while they’re open. You can collect this data in an observability backend in order to alert on the health of a pipeline. As first-class or community-driven approaches to emitting ‘in progress’ events proceed, I expect to see analysis and visualization tools come up with unique ‘timeline’ visualizations to help users understand pipelines.

To be ready for these enhancements, it’s important to have a good grasp of your observability data model. Decide what async work needs to be a part of a single trace, and which needs to be disjoint. Be sure to properly use tracers in OpenTelemetry to distinguish between “long running” and “short running” async workflows so that you can integrate unique span processors per pipeline. Be prepared to roll up your sleeves and get to work building some of this yourself, because it’s a genuinely difficult problem to solve in a generalized fashion.

Migrating to tracing

One of the biggest challenges in tracing isn’t technical, it’s organizational. Many organizations assign ownership of observability to some sort of operations, DevOps, SRE, or Platform Engineering team. These teams are responsible for rolling out observability across a wide variety of services, many of which are unmaintained (often because they don’t need active maintenance). This introduces friction. External observability teams don’t have the ability to modify the underlying source code for a service in order to add in tracing support, or to replace legacy logging and metrics frameworks with newer OpenTelemetry-based instrumentation. These teams have responsibility without authority, which is never a comfortable place to be.

This is such an acute pain point that the solution space is extremely broad. The most common approach is ‘zero code’ instrumentation agents—external software or libraries that can be loaded alongside an existing application and add in the necessary instrumentation via monkeypatching or bytecode manipulation. This method is flexible, but comes with increased overhead—not to mention increased complexity in data management, as configuration-based approaches or pipelines must be used to modify the data coming from an instrumented system.

The Extended Berkeley Packet Filters (eBPF) subsystems have emerged as a popular choice for instrumentation as well, as they allow for traces to be generated and propagated without burdening the application runtime and work in statically-compiled languages like Go. However, eBPF based solutions face challenges as well—a persistent one being their inability to cope with services deployed using mTLS. Another common challenge with eBPF is that while it can be lightweight, it is not free, and multiple eBPF probes can not only introduce system level latency, they can also conflict with each other. Care and coordination is required to deploy these tools without conflict.

I would love to tell you that there’s a magic bullet here, but there really is not. You will have to make decisions about what you really care about, and invest appropriately. You will need to employ a mix of strategies—agents to extend traces through unmaintained services, direct integration into new services, etc.—in order to propagate context throughout a system. You will have areas with less visibility, and areas with more visibility. Instrumenting for observability is part of developing and maintaining software, and there’s no handwavey eBPF agent that’s going to change that.

Where I do see a lot of potential is in AI. While it may be untenable to dedicate a team of humans to go through and update existing services to work with distributed tracing and OpenTelemetry, it’s a perfect task for an AI. This isn’t to say you can throw your codebase into Copilot or Claude and have it solve all your problems today—but I think a day will come where you’ll be able to teach a model about your existing logging patterns and have it update those legacy logs and metrics into unified, trace-first, telemetry.

Tracing as part of your data strategy

To wrap up, let’s discuss the biggest change that tracing is driving: a change of mental models. Historically, developers have thought about tools and workflows rather than data. When you ask someone what they do to understand what’s happening in production, what do they say? Most often, they’ll start talking about a tool. “I check the logs in Splunk,” “I go look at Dynatrace,” “I open a Grafana dashboard.” This makes a certain amount of sense.

The new approach is about making the implicit explicit. Rather than thinking “I’m just going to log some stuff and go find it later,” developers are taking ownership of observability as a part of their development process. This isn’t a ‘nice to have,’ it’s practically a requirement, especially in the age of AI. The more code that gets generated, the more we need observability baked into our systems. The biggest reason? Ironically enough, cost. It’s far cheaper to spend the extra time up front to think about the signals that your service emits and to create the right ones than it is to spend a bunch of extra time and money cleaning up that data on the backend.

Leveraging unified telemetry data improves your resilience, as well. Developers can add instrumentation as they go, then utilize that data in multiple ways. Rather than having different data consumers load down a service with different agents or libraries, OpenTelemetry provides a single standard that everyone can derive the insights they care about from. Product owners can turn traces into user journeys, security teams can feed everything into SIEM products, developers can get low-level data from production through continuous profiling, etc. Traces bind all of these together, providing the shared context that allows users to pivot between signals and get the full picture of what’s happening in a system.

Preparing for this change is both paradoxically easy and hard. It requires a culture shift more than anything else. Your developers may not want to adopt a new strategy, especially one that may not seem to have immediate payoffs. A wrinkle here is that these strategic shifts often happen in isolation. Because of the aforementioned organizational issues, the people who own the observability strategy are probably isolated from the people who have to change how they’re working.

In this situation, think holistically. You’re not going to convince someone to change how they work without also showing them what they’ll get out of it. Impressive backend improvements to data retention or query speed aren’t necessarily motivating to line developers, either.

Paradoxically, the UI and UX are the most important part of changing your data and mental models. Be sure to consider these factors when you’re rolling out OpenTelemetry, and leverage its vendor-agnostic data to not only evaluate alternatives to your current observability tools, but also to build bespoke visualizations as needed for your business logic.

Conclusion

If I got anything across in this paper, it’s that this isn’t simple. I would go so far to say that there are no simple problems in observability, period! The amount of complexity involved in even a small application can be overwhelming if you’re running that application at scale. Managing this complexity is at the core of what we do every day as software engineers.

Just because something isn’t simple doesn’t mean it’s impossible, though. It’s not impossible for us to understand the systems that wend through the world today. As a matter of fact, it is the triumph of our society that we have built tools to help us understand and conceptualize those complex systems. We’re able to feed eight billion people, after all! That’s pretty impressive!

Conceptualizing complex systems is, ultimately, what observability is all about. A complex system without observability is a dangerous system. It is, fundamentally, unsafe. Without models, we scratch along in the darkness, unable to predict the outcome of decisions. While we can always elect to ignore what our tools tell us, it is a far grimmer fate to not have the tools at all.

Understanding your own complex system may not be simple, either, but I would argue that it is crucial. Building reliable, resilient, and safe systems isn’t just a goal—it’s an element of craft. It’s part of the calling we have as software engineers. It’s an investment, not only in ourselves, but in those who come after us. It’s a covenant with our users and peers, that we care enough to explain what’s going on. It’s a promise to our business, that we aren’t mysticism-shrouded tech-priests getting fat off largesse. It’s the ability to answer questions, with confidence rather than vibes.

And if none of those work for you, it’s the ability to sleep soundly when you’re carrying the pager, because at least you know that you’ll be able to find and fix whatever’s wrong without losing your entire weekend.