Get full-fidelity visibility without full-volume costs with our Telemetry Pipeline update.Learn more

Login Start For Free Book a Demo

Debugging

The Guide to Kubernetes Debugging

April 30, 2025

Rox Williams

The Guide to Kubernetes Debugging

Kubernetes is widely used for deploying, scaling, and managing systems and applications and is an industry standard for container orchestration. Google engineers originally developed Kubernetes as an open-source project. Its first release was in September 2014, and since then, it has matured into a graduate project maintained by the Cloud Native Computing Foundation (CNCF). With the complexities of scale and distributed systems, debugging in Kubernetes environments can be difficult.

Troubleshooting CORS Errors in Offsite API Calls

December 19, 2024

Ken Rimple

Troubleshooting CORS Errors in Offsite API Calls

In this article, I’ll clear up the confusion around Cross-Origin Resource Sharing (CORS), specifically discussing preflight requests, which are the bane of engineers everywhere.

Debugging Kubernetes Autoscaling with Honeycomb Log Analytics

October 21, 2024

Liz Fong-Jones

Debugging Kubernetes Autoscaling with Honeycomb Log Analytics

Let’s be real, we’ve never been huge fans of conventional unstructured logs at Honeycomb. From the very start, we’ve emitted from our own codestructured wide events and distributed traces with well-formed schemas. Fortunately (because it avoids reinventing the wheel) and unfortunately (because it doesn’t adhere to our standards for observability) for us, not all the software we run is written by us. And Kubernetes is a prime example of such a load-bearing part of our infrastructure.

Always. Enable. Keepalives.

August 26, 2024

Lex Neva

Always. Enable. Keepalives.

As part of our recent failure testing project, we ran into an interesting failure mode involving the OpenTelemetry SDK for Go. In this post, we’ll show you why our apps stopped sending telemetry for over 15 minutes and how we enabled keepalives to prevent this kind of failure from happening in the future.

Software Engineering

A Practical Guide to Debugging Browser Performance With OpenTelemetry

December 1, 2023

Purvi Kanal

A Practical Guide to Debugging Browser Performance With OpenTelemetry

So you’ve taken a look at the core web vitals for your site and… it’s not looking good. You’re overwhelmed, and you don’t know what change to make because everything seems like too big of a project to make a real difference. There are so many measurements to keep track of and the standards cited seem even scarier. This is extremely normal. Web performance standards can feel impossible to meet for a lot of us.

Errors Got You Down? Honeycomb and OpenTelemetry are Here to Help

May 8, 2023

Valerie Silverthorne

Valerie Silverthorne

Errors Got You Down? Honeycomb and OpenTelemetry are Here to Help

It’s 5:00 pm on a Friday. You’re wrapping up work, ready to head into the weekend, when one of your high-value customers Slacks you that something’s not right. Requests to their service are randomly timing out and nobody can figure out what’s causing it, so they’re looking to your team for help. You sigh as you know it’s one of those all-hands-on-deck situations, so you dig out your phone and type the “going to miss dinner” text. The team places an order for takeout (a small perk of staying late: expensing dinner) and digs into what’s happening.

Visually Debug Your Systems With the Service Map Sandbox Scenario

February 6, 2023

Rebecca Carter

Get the Big Picture: Learn How to Visually Debug Your Systems with Service Map—Now Available in Sandbox

Honeycomb’s Service Map gives you the unique capability to filter by both services and traces so you can generate maps that isolate very specific areas of your environment. This helps you surface issues quickly and gain a granular understanding of how everything in your system works and evolves over time. And now that a Service Map scenario is available in our Sandbox, you can see it for yourself.

Product Updates

Solving a Murder Mystery: The Columnar Datastore Bug

December 7, 2022

Guest Blogger

Solving a Murder Mystery

Bugs can remain dormant in a system for a long time, until they suddenly manifest themselves in weird and unexpected ways. The deeper in the stack they are, the more surprising they tend to be. One such bug reared its head within our columnar datastore in May this year, but had been present for more than two years before detection.

Incident Response

Incident Review: Shepherd Cache Delays

October 18, 2022

Fred Hebert

Incident Review: Shepherd Cache Delays

In this incident review, we’ll cover the outage from September 8th, 2022, where our ingest system went down repeatedly and caused interruptions for over eight hours. We will first cover the background behind the incident with a high-level view of the relevant architecture, how we tried to investigate and fix the system, and finally, we’ll go over some meaningful elements that surfaced from our incident review process.

Debugging Just Got Faster and Easier With New Enhancements to BubbleUp

October 17, 2022

Fahim Zaman

Debugging Just Got Faster and Easier With New Enhancements to BubbleUp

We’re thrilled to share that we’ve made significant enhancements to BubbleUp that will enable users to BubbleUp from more parts of the Honeycomb UI beyond heatmaps. This means users can leverage BubbleUp’s powerful pattern detection on specific groups of users or conditions beyond numeric values, such as users from a particular region, using specific devices or operating systems that are experiencing a particular error message—or are using a unique part of your application, such as a discount code. This ability dramatically accelerates the debugging workflow because you can investigate so many more combinations of factors by asking a wider range of questions about your data to uncover underlying issues in way fewer steps, all from the same UI.

Product Updates

SLO Incident Review

September 9, 2022

Fred Hebert

Incident Review: Working as Designed, But Still Failing

A few weeks ago, we had a couple of incidents that ended up impacting query performance and alerting via triggers and SLOs. These incidents were notable because of how challenging their investigation turned out to be. In this review, we’ll go over interesting patterns associated with growth, and complex systems—and how these patterns challenged our operations.

Authors’ Cut – Build vs Buy

July 22, 2022

George Miranda

Authors’ Cut—Debugging with the Core Analysis Loop, and What to Build vs Buy

In this blog, we’ll dive into debugging with the Core Analysis Loop, the functional requirements for a backend datastore that make this possible, and whether it makes sense to build or buy your own observability solution.

We learn systems by changing them

July 6, 2022

Jessica Kerr

We Learn Systems by Changing Them

In the social world, there is no outside: we participate in the systems we study. I’ve noticed this in code: when I come to an existing codebase, I get a handle on it by changing stuff. Change some text to “HELLO JESS” and see it on the screen. Break something on purpose and see what fails. Refactor and find compilation errors. After I make some real changes and shepherd them into production, then I’m integrated with the code, I come to an understanding of it.

Why Intuitive Debugging Has Stopped Working for You

January 3, 2022

Pete Hodgson

Why Intuitive Debugging Has Stopped Working for You

It’s harder to understand and operate production systems in 2021 than it was in 2001. Why is that? Shouldn’t we have gotten better at this in the past two decades? There are valid reasons…

Incident_Report-The_Missing_Trigger_Notification_Emails_THUMB

December 17, 2021

Steve Lewis

Incident Report: The Missing Trigger Notification Emails

On November 18, between 00:50 and 00:56 UTC, an update was deployed that improved Honeycomb’s business intelligence (BI) telemetry available from our production operations environment. Contained within that update was a defect that escaped…

Incident Response

Software Engineering

blog_Do_You_Remember_20_Fires-THUMB

November 10, 2021

Fred Hebert

Incident Resolution: Do You Remember, the Twenty Fires of September?

From September to early October, Honeycomb declared five public incidents. Internally, the whole month was part of a broader operational burden, where over 20 different issues interrupted normal work. A fraction of them had…

Software Engineering

blog_game_launches_thumbnail

November 9, 2021

Liz Fong-Jones

Game Launches Should Be Exciting for Your Players, Not for Your LiveOps Team

This blog was co-authored by Amy Davis. The moment of launching something new at a game studio (titles, experiences, features, subscriptions) is a blockbuster moment that hangs in the balance. The architecture—distributed and complex,…

Software Engineering

AWS-lambda-debug-blog-image-v2

May 24, 2021

George Miranda

Easily Debug Your AWS Lambda Functions With Honeycomb

With the Honeycomb extension for AWS Lambda, you no longer need to make your Lambda functions Honeycomb-aware. Today, AWS announced the general availability of AWS Lambda Extensions, which make it easy for us to…

January 14, 2021

Kelly Gallamore

Kelly Gallamore

Community wins for the o11ydays!

The observability community helped us pay it forward for the o11ydays! Read about the contest winners and Honeybees’ favorite responses.

number 3

October 27, 2020

Danyel Fisher

Handle Unruly Outliers with Log Scale Heatmaps

We often say that Honeycomb helps you find a needle in your haystack. But how exactly is that done? This post walks you through when and how to visualize your data with heatmaps, creating a log scale to surface data you might otherwise miss, and using BubbleUp to quickly discover the patterns behind why certain data points are different.