The Director’s Guide to the Future of Observability: AI, OpenTelemetry, and Complex Systems

“All, everything that I understand, I understand only because I love.” — Leo Tolstoy

The 21st century has been defined by software, and by those who are responsible for it. The rapid pace of innovation in software design, development, and deployment has enabled teams to build faster, more responsive, and more differentiated software than ever before.

However, this innovation has not come for free. Every abstraction we build, every platform we adopt, and every dependency we rely on increases the complexity of our software systems. It is this complexity that forms the core of the greatest challenges facing engineering teams, and software organizations, today.

Conway’s Law states roughly that software design mirrors the structure of the team that builds it. First set forth in 1967, we can see and feel the impact of this observation on the software that we build—and use—every day. The days in which a handful of engineers could think, and reason about an entire software system are long past; we have traded away broad ownership for deep understanding of individual components. This specialization benefits our velocity, but adds yet another layer of complexity to understanding and diagnosing the system when things go wrong.

AI is accelerating this trend. As agents become active participants in the software development lifecycle—writing code, proposing changes, even deploying services—the authorship of our systems is becoming distributed not just across teams, but across humans and machines. Conway’s Law gets stranger when some of your “team” is an agent that doesn’t carry institutional context from one session to the next.

It’s no surprise that managing this complexity has become the single biggest priority of software organizations. There are a lot of words for how we do it: DevOps, Agile, Chaos Engineering, Site Reliability Engineering, Continuous Delivery, Extreme Programming—the list goes on. The fundamental nature of all these pursuits, though, is that they seek to do two things:

1. Increase the rate of change in a system
2. Reduce the amount of ‘bad’ changes in a system

Observability is another one of these words, but I believe that it should not be set alongside other techniques and philosophies; it should underpin them. Observability—a practice that provides teams with a way to ask fact-finding questions of their data, pursue leads, and generally explore anything and everything that’s occurring during an incident—is a foundational requirement for all of these pursuits, and without it, you will invariably fail.

Observability is the foundation of complex systems

Modern cloud-native architectures and decentralized or distributed systems introduce overwhelming amounts of complexity into software systems. Fundamentally, these systems are characterized by a few unique traits:

• Complex systems are reactive. They respond automatically to changing conditions and requirements based on traffic volume, business logic, or end-user demand.
• Complex systems are dynamic. They run on many nodes and runtimes and are managed by orchestration systems.
• Complex systems are cross-functional. They are built and operated by different teams working together, loosely coupled, all working at their own cadence.

The only constant in a complex system is change, and the management of that change—at scale—is one of the fundamental challenges put before engineering leaders today. In order to rapidly deliver value to our end users, we cannot rely on traditional methods of managing these changes, such as change management boards, lengthy pre-production review, laborious integration and acceptance testing, or reams of documentation.

It’s not that any of those things are bad, but they simply do not scale to meet the expectations of the market. New and emerging technologies further exacerbate this issue. How do you unit test generative AI when the same prompt may return slightly different results? How can you set up a validation process for decentralized systems when your code may be running on a huge variety of nodes across the world, all communicating on open protocols, but with no direct control over the environment?

The answer, then, is that we must adapt our processes and tools to satisfy these new requirements. If we cannot rely on old methods, we must change them.

Observability provides a path forward. It offers a different path towards understanding and managing this change, because it promotes a holistic approach to understanding complex systems. With observability, you can ship quickly and with confidence, knowing that the results of change are immediately visible to your teams. When incidents occur, you won’t spend precious minutes tracking down a particular dashboard or stakeholder in order to find crucial debugging data. Your teams are empowered to build better services, faster, and with more independence from slow and error-prone manual review processes.

The building blocks of observability

It’s probable that you’ve heard about observability, or even have an observability practice at your organization. The word has become extremely popular over the past few years, and this popularity has introduced a significant amount of confusion in the marketplace.

Some people claim that observability is indistinguishable from monitoring, and point to their log centralization and alerting systems as proof. Others swing to the left, arguing that observability requires continuous kernel-level inspection of network and system calls.

These arguments belie the truth that observability is mostly a practice. It has technical requirements, and these requirements do have material impacts on the efficacy of your practice. That is true. The real value of observability, though, is realized through organizational improvements and customer delight. Observability is, if you’ll permit some hyperbole, an expression of love. Not the kind of romantic love that we like to think about from TV, movies, or literature. No, it is about love for your end users, love for your team, and love for your organizational health.

This love results from the shared values and language that observability brings to reliability and performance conversations in an organization. Rather than endless fighting between managers and teams over vanity metrics, unclear ownership and accountability for dissatisfied end users, and burnout-inducing incidents and on-call schedules, observability promotes a consistent and clear set of measurements about performance and reliability, rich context and defined relationships between services, and a stream of interrelated data with enough depth to make identifying contributing factors to incidents and performance anomalies an easy and smooth process.

Observability is not built on three pillars of data, but three layers:

• Ubiquitous, high-quality telemetry data.
• Context-aware and high-performance data stores.
• Unified and shared understanding of what matters, as defined by Service Level Objectives and other observability measurements.

Let’s discuss each of these briefly.

Telemetry data

The fundamental signal in telemetry is the distributed trace, which is a logical representation of a single request and all service calls involved in its processing. Much like a stack trace records each call involved in an exception, a distributed trace allows you to see each service that a request called during its execution. Traces are made up of spans, logical events that represent some unique unit of work. In many ways, you can think of a span as a structured log.

Where spans differ from logging, however, is that they provide hard context. Each span has a unique identifier inside a trace, and each trace has a unique identifier from all other traces. What does this mean? In short, you can easily correlate events in a distributed architecture with each other, then analyze those events in aggregate. Traces allow you to ask fundamentally different questions about system performance than just metrics and logs. For example, “For all of our mobile users that are on the latest version of iOS, how many of them experienced a slow checkout, grouped by region?” or “Of the top ten slowest database queries, what do they have in common?”

This sort of aggregate analysis is extremely powerful, and not only greatly reduces the amount of time it takes to find interesting and valuable insights in your data, but is also crucial for building observability measurements like SLOs.

Beyond traces, there are other important signals: time-series metrics, logging, and profiles, to name a few. An observability practice will use them all in order to create effective and cost-efficient measurements. What’s important is less the type of signal, and more that those signals can be correlated with both hard context and soft context: shared metadata that’s consistent across signal types, cloud providers, application runtimes, and so forth.

You can think of these signals not as independent pillars, but as a braid of data that provides strongly correlated and interconnected points, giving you the full landscape of what’s happening in your system at any given moment.

As AI agents become more involved in writing and deploying code, this braid becomes even more critical. Custom instrumentation—attributes that describe the business intent behind a service call, the customer segment it serves, the feature flag it’s gated behind—isn’t optional in an AI-assisted world. It’s how you communicate context to agents that lack institutional memory, and how you verify that agent-authored changes are doing what they’re supposed to do in production.

Context-aware data stores

Historically, monitoring tools divide data into different analysis, query, and storage experiences based on the type of telemetry signals received. Full-text search for logs, dense time-series dashboards for metrics, and single-trace waterfall views for traces. This split makes sense if you think of these signals as independent pillars. However, as discussed in the prior section, observability requires correlation and context. Splitting data into separate stores and workflows causes friction, fragmentation, and isolated views into what’s happening in your system.

Telemetry is convertible between different formats. For example, many people monitor the health of web servers such as NGINX or Apache by converting access and error logs into metrics and throwing away the log messages themselves. Traces, as well, can be represented as metrics; metrics can be turned into derived logs or spans.

Underlying all of this is the fact that each of these point-in-time signals represents some sort of arbitrarily-wide event. Events are a powerful way to reason about telemetry data, as they free you from disconnected query and analysis experiences. Rather than having multiple query languages, visualization styles, and incompatible alert syntax, a single event-based data store allows you to create SLOs, alerts, and visualizations based on your objective rather than the initial data format.

Column-based data stores, similar to those used for OLAP purposes, are the perfect fit for recording and querying telemetry. They provide lightning-fast queries over trillions of events, and allow you to efficiently scale your ingestion based on demand. Even more valuable is the lack of cardinality explosions, a common woe for time-series metrics databases like Prometheus.

Cardinality refers to the number of unique values for a given attribute. In a traditional TSDB, each unique combination of attribute values (aka labels) produces a new copy of that series. Trying to measure per-customer performance or pod-level metrics is a recipe for disaster without the ability to handle infinite cardinality through column stores.

Speed matters here, too, and not just for humans. AI agents working through an investigative loop need sub-second query responses in order to be effective. An agent that has to wait thirty seconds for a query result is an agent that burns tokens, loses context, and produces worse outcomes. The same query performance that makes your engineers faster in an incident makes your AI agents viable as collaborators.

Finally, column stores accelerate adoption across teams. Many organizations are forced to grapple with tool sprawl—different teams use different things. In an incident, when minutes matter, do you want humans and AI agents grappling with a patchwork of query semantics across different tools, or using a single fluent language to find data regardless of its type? A unified query experience means your engineers can read and understand what an AI agent just asked your data store, and the agent can parse and build on queries your engineers wrote last week. Having everyone—human and machine—on the same page in terms of query semantics, visualizations, and alert types reduces communication errors and delays, and ensures your frontend, backend, platform, and infrastructure engineers use a common tongue.

Observability primitives and workflows

The final layer of an observability practice is what you actually do with the data. Rich, wide events being stored in a columnar format enables exciting and innovative visualizations and correlation functionality.

They enable you to build SLOs, a way of setting expectations about service availability and performance which is tied to business objectives rather than arbitrary performance counters. They can be inspected using heatmaps, which help you understand the shape of performance and visually spot outliers. Finally, they can be synthesized into service maps, helping you understand the connections between services in a system.

Heatmaps, service maps, and SLOs are the foundation of observability workflows. These three components give you actionable visualizations, a shared context for the shape of your system, and a shared language about what is valuable in terms of performance. It doesn’t make sense to spend all your time optimizing around arbitrary metrics when you can instead focus on end user journeys and other things that provide real monetary business value—a crucial consideration in today’s macroeconomic climate.

These components break you away from traditional ‘query and dashboard’ models of monitoring. Rather than spending time paging through endless, repetitive, and poorly-documented dashboards for your services during an incident, you can ask immediate questions to pinpoint the contributing factors to a failure. Rather than battling with other teams over their actual availability, you can use SLOs to codify the scenarios that are truly important to the organization and share them beyond engineering.

SLOs form an integral part of an observability practice as they provide inputs to not only the “Is it working?” question that is so commonly asked, but the equally important “How is it working?” question. They open up a new front of performance and reliability work, one that relies on the actual performance of your production environment and your team’s capacity to change it. Ultimately, these components lead to not just a change in how you run software, but a change in how you build it:

Observability can open up new, near-real-time debugging experiences in production. It provides a link between user analytics and performance data, informing data-driven decisions about what work to prioritize in order to support revenue growth. It empowers individuals and teams to solve problems by inspecting the true state of a system, rather than sit in the twilight darkness of half-remembered advice from a legacy engineer about how things should work. It directly impacts release velocity, increases end-user satisfaction, and reduces downtime.

Observability and OpenTelemetry

The first layer of observability is ubiquitous, high-quality telemetry data. Historically, this has been a challenge for teams to implement. While there are many solutions, there has never been a universally accepted and agreed-upon format for creating and expressing telemetry data across the wire. This has led to a variety of negative outcomes, such as an overreliance on proprietary agents, a huge amount of automatically generated data, and many open-source projects falling back to basic forms of telemetry like logging.

Thankfully, a solution exists: OpenTelemetry. Created by the merger of two major open-source observability projects—and backed by organizations such as Microsoft, Datadog, Splunk, Cisco, Honeycomb, and more—OpenTelemetry has achieved remarkable success and growth since its inception. Today, it is in production at thousands of organizations around the world, providing telemetry data to engineering and platform teams.

The core of OpenTelemetry is the notion that for cloud-native software to achieve observability, it must have built-in telemetry. To accomplish this, it provides a complete end-to-end system for creating, collecting, and exporting traces, metrics, and logs from applications and infrastructure. This data is processed according to a set of semantic conventions, which ensure consistent and clear metadata regardless of runtime or execution environment, host platform, or cloud.

In addition, OpenTelemetry is vendor-agnostic. You’ll never have to rewrite your instrumentation in order to switch analysis systems. It’s supported in over 13 languages—from Java to Go, to C++ and many more—in both backend and frontend applications. It’s also only getting better, as OpenTelemetry is the second most active project in the Cloud Native Computing Foundation, right behind Kubernetes. With over 2000 developers representing 200 organizations contributing, OpenTelemetry’s community is extremely strong. Honeycomb has been a part of the OpenTelemetry story from the beginning, contributing to the project’s shared mission of making observability a built-in part of cloud-native systems.

OpenTelemetry provides a single, standards-based, broadly supported and thriving platform on which to build your observability practice. Using OpenTelemetry’s broad array of existing integrations, you can get started in minutes. You can even adapt existing logging-based telemetry systems to function in OpenTelemetry through its bridges to legacy tools.

The real value, though, is when you combine out-of-the-box integrations with frameworks like gRPC, or service meshes such as Istio or LinkerD and your own custom instrumentation. Developers can use the OpenTelemetry API to write custom spans or metrics that describe the relationships between different components or services in a system, and provide rich, soft context by building new attributes that correspond to important business values.

In a world where AI agents are both writing code and querying telemetry, standardized instrumentation becomes a shared language not just between human teams, but between humans and machines. An agent investigating a production incident needs the same context-rich data that a human does—and OpenTelemetry ensures that context is expressed consistently, regardless of who or what wrote the code that generated it.

There are several unique use cases where OpenTelemetry shines:

Migrations

OpenTelemetry is the cornerstone of successful cloud or microservice migrations. Instrumenting a legacy application and the migrated components provides at-a-glance understanding of how the migration is impacting overall application performance.

SLOs

More and more organizations are adopting Service Level Objectives as a cross-functional measurement of end user satisfaction and engineering performance. OpenTelemetry data is trace-first, making it the ideal signal to build SLOs on top of.

Kubernetes

OpenTelemetry natively integrates with the Kubernetes ecosystem, enabling the collection of crucial performance data from pods, nodes, and other objects. This data can then be associated with application telemetry, giving you the ability to pinpoint problems down to the pod level when they occur.

Modernization

Since OpenTelemetry is designed to exist as an abstraction over a composable SDK, you can use it to modernize an existing in-house telemetry system. Even better, this approach allows you to save on effort from re-instrumenting existing code, combining old and new into a single vendor-agnostic format.

This is only a small selection of use cases, but the rewards are clear. OpenTelemetry is the best foundation to build telemetry for cloud-native applications, and is a crucial component of an observability practice.

What if I don’t use OpenTelemetry?

Perhaps you already have some internal framework that emits structured logs with a lot of attributes and a shared correlation identifier for each request—it’s not unheard of. This data looks a lot like the traces that OpenTelemetry emits, maybe with a few gaps here and there. At Honeycomb, we don’t think that you should have to re-instrument your entire system to start seeing the benefits of observability, so we’ll gladly accept structured events as JSON to let you get started. You’ll be able to see the difference between traditional log viewers and context-aware observability tooling.

What if you want more, though? This is where OpenTelemetry comes into play. It’s extensible, customizable, and allows for progressive enhancement of your existing telemetry in order to take advantage of features built on the standard. For example, you can use tools like the OpenTelemetry Collector to easily normalize your pre-existing telemetry into newer formats, and automatically add in rich context from runtimes like Kubernetes. You can even swap out parts of the OpenTelemetry SDK, allowing you to use both your existing structured events alongside libraries and frameworks that work with OpenTelemetry out of the box.

Like we said earlier, the requirement for observability is high-quality, context-rich data. OpenTelemetry is a great way to get that if you don’t have it—but if you do, you can choose to use that data or make OpenTelemetry part of a progressive enhancement story for your observability practice.

Regardless of how you create the data, what really matters is that your engineers get into the practice of instrumentation: writing telemetry code that helps describe the intent of a service or system. You need to add custom attributes to identify important facets of your business logic or service workflows, such as customer identifiers. This metadata levels up observability visualizations like heatmaps and SLOs. It’s the difference between parsing through endless logs or too-broad metrics graphs and being able to pinpoint where and why incidents are occurring. OpenTelemetry is certainly a great way to do this, and provides a convenient way to normalize and standardize your telemetry data sources, but it’s not the only way.

Observability in the age of AI

The software development lifecycle is compressing. AI coding agents can produce in minutes what used to take days. Inference costs are dropping. The barriers to creating software—writing it, shipping it, iterating on it—are lower than they’ve ever been and will keep falling.

This has different implications depending on what kind of organization you are. For companies whose primary product is software, this is a competitive question worth taking seriously. When code is cheap to produce, the moat isn’t in writing it—it’s in knowing whether it works, whether your customers are happy, and whether the system is doing what you intended. For organizations where software is a value accelerator rather than the product itself—banks, retailers, insurers, logistics companies—the calculus is different but the conclusion is similar. A great mobile app might be why a customer chooses your bank over a competitor, but the app isn’t the bank. What matters is that it works, that it keeps working, and that you know when it doesn’t.

In both cases, the verification function is becoming the bottleneck—and the differentiator. The traditional SDLC was organized around the assumption that writing code was expensive and slow, so we built elaborate processes to prevent bad code from reaching production: code review, integration testing, acceptance testing, change management boards. These processes still have value, but they were designed for a world where changes happened at human speed. When agents can propose, implement, and ship changes faster than a human can read the diff, the center of gravity shifts. Tests tell you whether code conforms to a specification. Observability tells you whether the software actually does what it’s supposed to do for the people using it.

This is the distinction that matters. Tests verify intent against specification. Observability verifies behavior against reality.

That distinction has always been important, but it becomes critical when the volume and velocity of change outpace your ability to review it upfront. Observability becomes the verification layer—the thing that closes the loop between “we shipped a change” and “that change is doing what we expected in production.”

Agents as authors and collaborators

AI agents are increasingly involved at every stage of the software lifecycle. They write code, propose architecture changes, generate tests, deploy services, and investigate incidents. This is already happening at organizations of all sizes, and it’s accelerating.

This creates a new dynamic for observability. In the traditional model, the people who built a service were the same people who understood its intent and could reason about its behavior in production. With agents in the mix, there’s a gap: the agent that wrote the code may not be the same agent—or human—investigating it when something goes wrong. The institutional knowledge that used to live in someone’s head now needs to be encoded somewhere durable. Telemetry data, enriched with custom attributes that describe business context and intent, becomes that durable record.

Observability also becomes the mechanism through which agents verify their own work. An agent that deploys a change and then checks SLOs, examines heatmaps, and queries trace data to confirm the change behaved as expected is operating in a tight feedback loop—the same kind of loop that makes human engineers effective. Agents that can access observability tooling through protocols like MCP (Model Context Protocol) can participate in investigations, surface anomalies, and compile reports for human review. They become collaborators, not just tools.

This works in both directions. Humans use observability to verify what agents are doing—to understand how agent-authored changes are affecting the system. Agents use observability to verify their own outputs against production reality. The shared layer is the same: rich, context-aware telemetry data, stored in a system that’s fast enough to keep up with the speed of AI-driven iteration.

What this means for your practice

The implications for engineering leaders are concrete. First, instrumentation becomes a form of communication—not just between teams, but between humans and agents. The custom attributes you add to your telemetry are how you teach agents what matters about your system. Skimping on instrumentation has always been a false economy; in an AI-assisted world, it’s actively self-defeating.

Second, SLOs take on new importance. In a world where changes are being proposed and shipped at machine speed, SLOs become the contract between human intent and system behavior. They codify what “working” means in terms that both humans and agents can evaluate. An SLO violation triggered by an agent-authored change is a clear, actionable signal—far more useful than trying to review every change before it ships.

Third, speed becomes non-negotiable. AI agents working through an investigative loop will generate, test, and discard hypotheses faster than any human. If your observability tooling can’t keep pace—if queries take thirty seconds instead of sub-second—the agent loses context, burns tokens, and produces worse results. The same query performance that helps your engineers in an incident is what makes AI agents viable as partners.

Finally, the organizations that will get the most out of AI are the ones that already have strong observability fundamentals. AI is an accelerant: it amplifies what’s working and compounds what isn’t. If your teams lack shared context, if your telemetry is sparse and inconsistent, if your tooling is slow and fragmented—AI won’t fix those problems. It will make them worse, faster.

Conclusion

In this challenging macroeconomic environment, the demands to do more with less have never been clearer. This challenge does not simply mean we can deprioritize reliability, or systems understanding. No, the costs to reputation, customer satisfaction, and the well-being of our teams have never been higher.

Observability promises a solution to these Gordian Knots: a rethinking of our traditional model of monitoring towards a more collaborative, open, and aligned way of operating complex systems at scale.

While the challenges have never been greater, there are significant rewards for organizations who can meet them. Among these rewards are improved customer retention, better alignment between business and technical operations, reduced monitoring costs, and fewer long-term incidents.

Observability is love—for your end users, team, and organization. That love now extends to a new kind of relationship: the one between the humans who build and run software and the agents that help them do it. Observability is how we maintain understanding and trust in systems we share authorship of. It’s how we stay connected to the reality of what our software is doing, even as the pace and means of creating it change around us.

It is that same love that we pour into Honeycomb—because we care about your end users, team, and organization.