Last week, we sat down with the authors of Observability Engineering for a live AMA. We ended up getting so many questions (pre-submitted and live) that we couldn't get through them all.

Charity, Liz, George, and Austin kindly stuck around afterward to answer more, ranging from low-hanging observability fruits and telemetry to AI and what software engineers can do that Claude can't.

Missed the live session? Watch it on demand now:

Additional questions

What was the most painful thing to read from the first version of your book?

Charity: The predictions chapter at the end. And yes, if you ask me again in three years, I expect my answer will be the same.

George: We were still pretty passionate and maybe a bit raw from arguing across the industry about what observability meant and how it was different from monitoring. We relitigated those arguments repeatedly in several early chapters (that were written out of order—and what was in the first edition book was even after we edited our arguments way down). It felt like we needed to defend ourselves.

This time around, all of that has changed. The industry has started to catch on. It's less about the differences and more about what they enable. We included many more voices and viewpoints. We spent less time arguing and more time examining impacts across use cases. Now I think the book is more inclusive, way more streamlined, and yet somehow twice as meaty.

Why do we need so many metrics, logs, and traces when only a small portion of them are useful?

Charity: Is this a trick question? We don't need so many metrics, logs, and traces. We should think much more critically about the telemetry we collect, instead of treating it like it's carpet bombing or nothing. One arbitrarily-wide, structured log event or well-designed trace can replace hundreds of metrics and spammy logs, and be cheaper, more effective, and easier to reason about with its connective tissue intact. Stop making decisions about signal types at write time. Treat your telemetry like data.

George: What Charity said. Also, if you're asking that question, it sounds like you might need to sample more.

How do we get engineers to think for themselves again rather than responding to everything with “...well Claude says...”?

Charity: First, will you please tell me when this golden age of thinking for ourselves ever existed? It sounds exhausting to me. We are humans, we take shortcuts—it's one of our charms.

But I do have some practical advice for you: be specific. When you want someone's opinion, ask for their opinion. When someone gives you Claude's opinion, feign confusion. “Who is this 'Claude'? Alas, I do not speak French,” or “I have a Claude of my own, I want to know what Liz thinks.” But try to say this in good humor. After the initial shock at its capabilities, we are all well on our way to understanding the limits of AI. Stack Overflowification comes for all authority, in time.

How do you manage human-in-the-loop (especially domain experts) in observability?

Liz: Signal-to-noise has always been a challenge, but the robots are really good at picking out signal. It is our job as the humans to then make decisions about what to do with the signal.

George: I'm going to presume you're asking about the operating procedure vs. the orchestration layer (e.g., Temporal, Langgraph, etc). Even shifting to process instead of technology, the target keeps moving and best practices in AI are evolving. That said, check out the revamped “Getting Started with Observability Analysis” chapter in the book. Specifically, the section on Agentic personas.

We tried not to tie AI recommendations in the book to specific tools (writing about AI risks making your content obsolete before it even gets published), so we focus on process. The thing to remember though is that the copilot, commander, and caretaker personas mentioned are all human-in-the-loop processes; the only thing that changes is if the human gates before or after action. Which agent “persona” you decide to use will depend on the sophistication of your observability tooling, robustness of your orchestration layer, and how much trust your particular agents have earned. Generally speaking, most people will start with the copilot and work their way toward the caretaker.

When an AI application produces a wrong or unexpected response, how can observability help teams determine whether the problem came from the model, the prompt, the data, or the surrounding infrastructure?

Austin: That's the fun part—it can be all of those things at the same time! I've been doing a lot of thinking about this and there's two ways I'd slice it: there's a bundle of bog-standard stuff that you want to look at (golden signals, etc.) about the actual runtime environment, and then there's a lot of attributes that you'll want to build into traces. You're never gonna get the same thing twice out of an agent or AI application, so you want to shift as much of the attribution over to the client as possible and bake it into your spans.

Now, there are a lot of ways this can get complicated real quick (e.g., storing and querying prompts or outputs). I'm excited to see more progress on interesting ways to do annotation and classification closer to the generation loop itself.

I want to challenge the notion of “production” is the only place where you can actually see things. I understand and agree in general, but testing beforehand is so important that I'm not sure if it's the highest bit about which we should care.

Charity: Who said production was the only place you could actually see things? Certainly not me. On the contrary, if we relied exclusively on production for detecting faults, we would be swimming in far too many of them to pick out the long tail of aberrant behaviors.

Testing and validating pre-production is vital. Keeping up on improvements to the state of the art is necessary. But the mistake I see most software engineering teams making is investing too heavily in pre-production testing at the expense of their production tooling. And far too many companies still assume software engineers will never look at production at all. Until that changes, I will keep banging this drum.

What is the one advice you would provide a young person who is considering entering computer science today?

Liz: Focus on what you can uniquely do that Claude cannot. If the only value you provide is pressing enter or acting like the sipping bird from the Simpsons episode who always presses y, then why should anyone pay you? At the moment, the models do not learn over longer time horizons, become myopically focused on individual tasks, and do not have judgment about wider design calls. That is what you should specialise in, rather than the writing of the code itself.

George: Prioritize learning core fundamentals around system architecture, algorithms, and leveraging AI tools for code generation. There's less value in cranking out code itself and more value in architecting outcomes. The importance of systems thinking, turning that into system design, and knowing which data matters, how it moves, and how to apply it is what really excites me about the AI era. Technology is only valuable when it solves real customer or business needs, so the domain expertise and people skills matter more than ever. You will need to think bigger than ever before because, now, accomplishing the small things will increasingly become a commodity.

In the AI space, I think one of the durable questions is how to move as much into the deterministic layer that we give the agents as possible, and the right types of things. Do you agree and if so, what have you seen successfully move into determinism for instrumenting with AI? Sorry if this is in the book, not had a chance to read it fully yet.

Austin: I kinda agree and kinda disagree. I remember about a year and change ago, there was a lot of “well this would be good if the agents could talk to a language server, it seems so inefficient...” and then the models/harnesses just got really good at using grep and it turns out that it didn't really matter that much.

I think there's a lot of value in having models write code in order to achieve outcomes, such as refactoring. I think you discover those deterministic outcomes through a lot of non-determinism. If a task was easy to make deterministic, then we'd already be doing it that way! Ultimately I think that this is less of a binary than it is a sliding scale—common patterns will get baked in over time. For example, I recently refactored a library in our codebase that had a bunch of different span attributes created with strings (e.g., span.SetAttribute("foo", "bar")) scattered throughout. Some had the same key, some had subtly different keys. I had an agent go through, find all the callsites, analyze/classify the attribute keys, then refactor them all to a shared helper. What's really interesting and fun about doing this with AI is that it went ahead and just wrote a test case for the entire library that fails loudly if someone tries to set an attribute using a string rather than using the helper.

When a team starts the observability journey, what is the low hanging fruit to get started that will yield the most value?

George: Auto-instrumentation is a great place to get started and you'll get a ton out of the box. But custom instrumentation is where the value really accelerates. Check out Jeremy Morell's contributed chapter, Making Structured Events Arbitrarily Wide. There's a ton of concrete recommendations around standard environmental information to help you quickly ramp up value as you get started on your observability journey.

Conclusion

That's a wrap on this AMA, but if you want more time with the authors, Liz Fong-Jones is running a live masterclass, walking through how to actually put Observability Engineering into practice. Save your spot here, or watch this AMA on demand if you missed it.