They Aren’t Pillars, They’re Lenses

To have Observability is to have the ability to understand your system’s internal state based on signals and externally-visible output. Honeycomb’s approach to Observability is to strive toward this: every feature of the product…

Reducing Build Time with Observability in the Software Supply Chain

Using Honeycomb to remember to delete a feature flag

Feature flags are great and serve us in so many ways. However, we do not love long-lived feature flags. They lead to more complicated code, and when we inevitably default them to be true…

Getting At The Good Stuff: How To Sample Traces in Honeycomb

(This is the first post by our new head of Customer Success, Irving.) Sampling is a must for applications at scale; it’s a technique for reducing the burden on your infrastructure and telemetry systems…

10 tech startups for 2020 that will help the world build more resilient, secure, and observable software

From “Secondary Storage” To Just “Storage”: A Tale of Lambdas, LZ4, and Garbage Collection

When we introduced Secondary Storage two years ago, it was a deliberate compromise between economy and performance.

Incident Report: Running Dry on Memory Without Noticing

On November 6, 2019, we intermittently rejected 1-3% of customer telemetry data at ingest for four periods of 20 minutes each. The trigger of the incident was a slow memory leak that manifested over…

OpenTelemetry vs OpenTracing | Understanding OpenCensus and Related Terms

This guide provides clarity on past and current projects, how they differ, and what you need to know to futureproof your observability stack.

SLOs Are the API for Your Engineering Team

So You Want To Build An Observability Tool…

I’ve said this before, but I’m saying it again: observability is not a synonym for monitoring, and there are no three pillars. The pillars are bullshit. Briefly: monitoring is how you manage your known-unknowns,…

I test in prod

Understand Your AWS Cost & Usage with Honeycomb

First published in August 2019. AWS bills are notoriously complicated, and the Amazon Cost Explorer doesn’t always make it easy to understand exactly where your money is going. When we embarked on our journey…

Observability — A 3-Year Retrospective

Treading in Haunted Graveyards

Part 1: CI/CD for Infrastructure as Code At Honeycomb, we’ve often discussed the value of making software deployments early and often, and being able to understand your code as it runs in production. However,…

Making Instrumentation Extensible

Observability-driven development requires both rich query capabilities and sufficient instrumentation in order to capture the nuances of developers’ intention and useful dimensions of cardinality. When our systems are running in containers, we need an…

Reflections on Monitorama 2019

This year was my third in a row attending (and now speaking at!) Monitorama. Because the organizers do a great job of turning introverts into extroverts for three days straight, it’s always a fun…

Investigating Timeouts With Tracing Using Sentry

Tracing is one of the key tools that Honeycomb offers to make sense of data. Over the last few weeks, we’ve made a number of improvements to our tracing interface — and, put together,…

Automating Collection of Troubleshooting Data with Triggers: a How-To Guide

Everyone wants to be more efficient — to spend less time on the tedious things, and more time on the things that move the needle. As much as possible, if you can automate those…

Sustainable Operations in Complex Systems with Production Excellence

Stop Your Database From Hating You With This One Weird Trick

Let’s not bury the lede here: we use observability-driven development at Honeycomb to identify and prevent DB load issues.