We recently wrapped up a large-scale, multi-month Kafka migration project. We used to run self-hosted Confluent Platform and ZooKeeper as clusters of AWS EC2 instances, and now all of our Kafka clusters run open-source Apache Kafka 4.1.1 running in KRaft mode and deployed to AWS EKS.

There are insights and lessons within the story of how we did this migration that are worth sharing. In this blog post, I highlight key themes that set this project up for success: why committing to learning from and building on past incidents and experiences matter in projects like this, why it’s important to design any kind of Kafka migration with rollback and safety top of mind, and how our sociotechnical processes support Kafka migration execution in ways that build teams’ confidence.

A note to our customers: this post describes work related to a May 7, 2026, scheduled maintenance event on our US instance. We missed the mark on timely, proactive communication to you about that event, and we sincerely apologize for the negative impact it had on you. We have a separate post detailing our response to the customer impact of the event, including how we will improve our processes to ensure we provide additional mitigations and support to prevent similar communication issues in the future. Because the engineering work to prepare for and execute this migration was extensive, we’ve chosen to address the external and internal aspects of this event in separate posts; the omission of the external perspective from this post is not intended to minimize the customer-facing impact.

Kafka’s purpose at Honeycomb

At Honeycomb, Kafka is the beating heart of our observability data ingestion pipeline. It sits between customer observability data arriving at our edge and services like Retriever writing that data to our columnar store, ready for customers to query in Honeycomb. Kafka has been a great fit for our needs because we have to move and process millions of observability events per second through our systems, and Kafka’s capabilities and guarantees afford us numerous critical benefits:

• Lets us decouple data ingestion from data processing in a way that allows us to update our core ingestion path services several times a day without causing downtime for our customers
• Provides strong data buffering, reliability, and durability guarantees to ensure customer data isn’t lost when we accept it
• Allows us to flexibly scale in order to meet growing ingest demand
• Gives multiple consuming services the ability to read the same data stream independently, so all of our services can be in agreement about the data we’re receiving

Why we migrated

Prioritizing a large-scale Kafka migration requires substantial cross-team and cross-functional alignment. Asking for resources across multiple teams for a project that we knew would take several quarters to execute is not a thing that can be wished for and hoped into existence. We had to be very clear about the problems we were solving and why this project aligned with business objectives. So what were our reasons?

First, there were technical capabilities we had to be able to prove as engineering developed Honeycomb Private Cloud. Installations of Honeycomb Private Cloud require Kafka to be running on AWS EKS as part of its packaged deliverable. The way we had been running Kafka in our SaaS infrastructure would simply not have worked for Honeycomb Private Cloud installations. For long-term maintainability and sustainability of operating Kafka, we had to be able to run Kafka the same way and prove that running Kafka on Kubernetes would be able to handle production scales of traffic. Additionally, it has become increasingly more viable to deploy and run Kafka on Kubernetes in production-scale environments (we use Strimzi for the orchestration and management layer).

Second, migrating Kafka presented an opportunity to better align our current internal operational patterns and, in turn, sunset older legacy patterns that our administration of Kafka depended on. Our Honeycomb services are built and deployed to Kubernetes, and we determined that we could use a lot of those same patterns for Kafka.

Finally, migrating gave us a level of deep operational control and flexibility that we wanted. For example, we were finding that the time to recover from our weekly practice of replacing Kafka brokers was gradually getting worse. A couple of years ago, this took 8 to 12 hours in production, and before the migration started, this took 48 to 72 hours. The issue centered on Confluent Platform’s Tiered Storage functionality, a solution that we depended on, but also a closed-source one that gave us no means to fix issues directly without support from Confluent. Migrating to open-source Apache Kafka gave us more agency to fix issues like this if we ever encountered them.

Functional requirements will constrain migration options

Deeply understanding the behavior of the services that depend on and interact with Kafka dictated our viable migration target options, as well as the means to execute the migration. One of the most impactful of these functional requirements came from Retriever, which consumes then writes the data coming out of Kafka into our columnar store. The way Retriever handles Kafka offset management precluded us from using common Kafka migration tooling options.

Retriever ingestion workers don’t commit their partition offsets back to Kafka, but instead maintain their offsets internally. Why is that? Because Retriever has strict semantic delivery requirements (it’s effectively the exactly-once semantic) and strict data guarantees it has to uphold; Retriever cannot consume a message it has already serialized and written to the columnar store. Precise offset tracking and management is critically important to us.

Additionally, as described in Chapter 13 of Observability Engineering, Second Edition, Retriever uses parallel ingestion workers to consume a single Kafka partition: one consumer uploads finalized segments to the columnar store and the other consumer spot-checks that the resulting segment data produced is identical between the pair. We operate our system as not just exactly-once, but exactly-twice (across the pair). This design matters because Retriever checkpoints its Kafka partition offset to its local disk, and all ingestion workers will use these checkpoints when Retrievers need to be restarted due to a deployment or when a Retriever node instance is replaced. Retrievers can resume consuming from the checkpoint on the backup it restores from when it comes back online. In contrast, using a pair of consumer groups would result in divergent offsets being committed.

This dependence on internally managed and precise offset management meant that cross-cluster mirroring solutions like Kafka’s MirrorMaker 2 were not appropriate for our use case, because of its use of offset translation mechanisms. If we had used this to replicate messages between clusters, Retriever would not have been able to use its checkpoints to reliably retrieve the next messages from their respectively assigned partitions.

It’s important to pay close attention to your Kafka dependencies’ functional requirements, as they will constrain how you can execute your migrations.

Non-functional requirements will constrain them too

We maintain a 99.99% ingest availability SLO and tune our systems to maintain very fast end-to-end ingest latencies. When you send an event into Honeycomb, we need to be available to accept it and you need to be able to query that event in Honeycomb within one minute. We are particularly sensitive to all sources of latencies throughout the ingestion path, including Kafka disk I/O latencies.

This means that we ruled out using some managed Kafka solutions that use diskless topics, like Confluent’s Warpstream, because we can’t trade higher latencies for more cost-effective data transfer and storage. We optimized our choices for the most performant, lowest latency storage solutions for fresh data. We determined that using an EKS instance’s NVMe instance store rather than high IOPS provisioned EBS volumes (like io2) guarantees the lowest possible read and write latencies. By choosing NVMe instance stores over EBS volumes, we accept another tradeoff: Broker replacement and data recovery will take longer when using NVMe instance stores because all of the buffered data has to be rematerialized from scratch. EBS volumes can be detached and reattached during replacement, speeding up this process, but incurring an ongoing cost in dollars and latency (there are no EBS savings plans).

Every implementation decision you make, like what storage approach to use for Kafka data, needs to be informed by your non-functional requirements. Any Kafka migration evaluation must factor all requirements, and they should constrain which options you can use. Making the right design choices early in the process will pay off when you need to design the safety mechanisms of migration execution.

Build on institutional knowledge and wisdom

When I joined Honeycomb in January 2025, a wealth of history, knowledge, and wisdom were documented and left for me to leverage. I came with my own style and opinions on how we might achieve our goals, but it was vitally important to build on my predecessors’ wisdom. I prioritized understanding why we had designed Kafka the way we did to inform how I wanted to proceed. When I put together my proposal and evaluated the best options we had, I arrived at the same conclusions my predecessors did. This alignment gave me the confidence that I was thinking about the problem space and the potential solutions correctly.

Making the effort to find documented knowledge and history will help inform your decisions when you’re considering a Kafka migration of your own.

Applying lessons from incidents pays dividends

Before we started our migration work, in December 2025, we had a major incident that affected one of our production Kafka clusters. This was a stressful incident for many of us, which required completely evacuating the Confluent Platform Kafka cluster that we were on. I paid close attention to the mechanics of the emergency evacuation, because if you look at it from the right perspective, evacuation kind of looks like steady-state migration, but under time pressure and duress.

I was part of the incident response team for that incident, and I participated in the subsequent incident review and public report. We learned something new about our architecture through the incident that we had speculated might be true, but had never actually tried to do. Retriever could reset its checkpointed partition offsets to zero, and then be pointed at a new Kafka cluster. Retriever would then resume reading messages from the beginning of the partition on the new cluster. We could do this deterministically and dynamically via a feature flag, without dropping data or losing data continuity.

Why was this finding meaningful? The experience of the incident opened up a Kafka migration path that we thought we could not do, but because of the outcome of the incident, we learned this was actually a viable path for us to consider. We could execute a coordinated sequence of cutovers without dropping any customer data while maintaining full data continuity.

This pathway was made possible because of the lessons we learned from the incident. Learning from your past incidents is vital because if you are able to learn from them, you can build on those lessons in your migration designs and turn a stressful process into a more robust, safe, repeatable set of procedures.

Seizing the opportunity to level everyone up

Running Kafka ourselves meant confronting all of the implications of vendor independence with Kafka. Deciding to do the migration meant we were accepting terms of operational responsibility. We had to make sure that the teams who would be interacting with and operating Kafka would have the right resources to do that. The knowledge and expertise had to be set up to scale; these things could not just live in my head.

It was just as important to execute the migrations as it was to create the bridges of scalable knowledge between the old and new ways of operating Kafka. It was an opportunity to build something internally with broad value: a pedagogical library resource within internal documentation that would teach Honeycomb bees about the fundamentals of Kafka, how Kafka fits into our architecture, and how to operate and interact with our new Kafka clusters. Committing to building out a reliable foundation of knowledge benefits everyone, whether they are brand new to Kafka or they are joining an on-call rotation that will be on the hook to operate Kafka. You don’t want to be left in a position where you’re scrambling to hire to fill a knowledge or expertise gap if you can help it.

Prioritizing rollback procedures

The functional and non-functional requirements that constrained our options combined with the lessons of the Kafka incident in December directed us to make sure that our migrations’ safety guardrails and rollback scenarios were robust. We pushed some of our earlier migrations out to ensure we got the safety guardrails and rollback steps right. The stakes were too high, and we could not find ourselves caught flatfooted if we had to stop and reverse course during a botched production migration. If we had to make a decision to roll back because something went sideways, we were going to be prepared.

We run one Kafka cluster in each of our environments, consisting of three tiers of environments in two different regions: Kibble, the lowest environment; Dogfood, the next tier up; and finally the Prod clusters—six clusters in total. We exercised our rollback design in one of our Kibble environments. We migrated it fully forward, then fully backward, then partially forward and backward with our rollback procedures, then finally, fully forward a second time. When we explicitly migrated backwards to prepare for the rollback test, that procedure took over four hours.

Kafka migrations like these are big time and resource commitments, but dedicating time to the practice of migration execution was crucially important to us to ensure we got the processes nailed down as much as possible. This was another commitment we had made to learn directly from our migration experiences and use that as a feedback loop back into our migration processes.

The practice of execution reduces uncertainty and creates learning opportunities

The original template of our migration procedure came directly from the emergency evacuation runbook we created during the December Kafka incident, which laid out the multi-team choreography and process of the evacuation itself. We followed the procedure and recorded the experience of every migration we did in a multi-tab document. For each migration—whether Kibbles, Dogfoods, or Prod clusters—we had a corresponding “What did we learn?” tab in the document. Whenever we encountered something new or weird during the migration, we added a bullet point to that tab in real time. We reviewed what we learned a day or two after each migration, and we used those lessons to refine the template for the next migration. This refinement feedback loop allowed us to do some really crucial things that greatly reduced the process’s uncertainty and created strong learning opportunities for us as we executed each one.

First, we defined very clear roles and responsibilities, and we optimized how our team coordinated execution. Running a migration resembled running an incident: We coordinated in real time on Zoom; we had a migration lead and a communication lead; and we assigned key runbook responsibilities to other people. But early migrations were also massive, multi-team efforts. The original migration required coordination across seven teams. Because we had multiple migrations to do, we didn’t want all seven teams to have to be present for every migration. So over time, our team practiced and gradually took ownership of running other teams’ runbooks for later migrations. Eventually, our team was the only team required to run the migration. We built our confidence by having each person on our team rotate what they did each time. All of us got firsthand experience running every critical piece of a complex and choreographed process.

Secondly, it created opportunities to surface confusing or ambiguous things or invalid assumptions about the procedure. We gained valuable experience documenting these things, and we refined our processes well enough to take our migration executions from four to five hours down to executions of two to three hours.

Finally, it allowed us to identify and define what telemetry we needed out of Kafka and Kubernetes to make sure we could explore how Kafka was behaving in real time. We created and deployed a telemetry pipeline consisting of OpenTelemetry Collectors deployed to the EKS clusters running Kafka, which scraped Prometheus JMX metrics, extracted information through the Kafka Admin API, and gathered Kubernetes node metrics and events. We sent all of that directly to Honeycomb. We built out tactical Honeycomb Boards that centered on key signals we needed to verify during the migration. We used these visualizations as checkpoints to continue or pause if something didn’t look right.

All of us gained confidence along the way, and none of us had to play hero when we had to troubleshoot things. We saw no clearer evidence of the growth of our team’s confidence and expertise when the team executed a full Kafka migration while I was on PTO.

My team saw their growth for themselves

I took a few days off a week before we did the production migrations. We had one remaining non-production migration to do. I considered pushing my PTO back to be there for the migration just as I had for all of the others, but my manager pushed back against this temptation and made me remember 1) I should prioritize my own care and 2) having the team do the last non-production migration without me in the room was a perfect opportunity for the team to see for themselves how far they had come.

Before this project, I was the de facto “Kafka expert” on the team. Throughout the course of the project, I had seen my team grow in confidence, in knowledge, and in comfort by virtue of them being directly involved with each migration’s design and execution. I am still there and available to answer deep Kafka questions, but the rest of my team can meaningfully do this too.

I arranged for the team to execute the last non-production migration while I was on PTO. I had every confidence they could do this migration without me there. Do you want to know how that migration went? They executed it flawlessly, with no emergent issues to troubleshoot whatsoever, and they set a record for the fastest turn-up. I was so immensely proud of the team for achieving that. It gave us the confidence that we were going to succeed when we executed the production migrations.

The full impact of generalizing migration procedures

Designing the migration processes for repeatability and generalizability were both important components of this migration. We now have the ability to use our migration procedure in the future for any number of migration modalities: self-hosted to managed, EKS-to-EKS cluster, evacuation and disaster recovery, etc. The overarching simplicity of the designs that make them generalizable come with some important tradeoffs worth weighing. We accept some reductions in consume availability for that conceptual simplicity, and that’s a deliberate choice because it is informed by what our services and systems tolerate.

I said earlier that we uncovered a new migration pathway during the December Kafka incident. Refining that evacuation procedure made under duress took a lot of time and effort to guarantee data safety and continuity for future migrations. The shape of the process’s design was dictated by what our services and systems could tolerate, and in our case, that meant the tolerances of services dependent on offset preservation. By cutting over the producers first, letting the consumers finish reading from the old cluster, followed by cutting over the consumers and resetting their checkpointed offsets, we end up creating a window of downtime between the producer cutover and the consumer cutover. We haven’t interrupted produce requests or lost any of our customers’ data, but fresh data won’t be seen by the consumers during that window and, in turn, customers will see that gap.

Guaranteeing data safety and continuity as well as accepting a window of downtime to achieve that has a material effect on what success looks like. A migration procedure that has been mechanically executed well does not guarantee a successful migration sociotechnically. As we mentioned at the beginning, how effective we are in communicating scope of impact and expectations to our colleagues and customers matters just as much as smooth mechanical execution. Succeeding in one does not guarantee success in other areas of impact.

Your mixture of sociotechnical system tolerances combined with what tradeoffs you’re willing to accept will likely differ. Projects of this scale need to account for as many of these impacts as possible.

You can do difficult things

Learning from your experiences, both historical and present, is essential to build on when you undertake a project of this magnitude. If Kafka is as vitally important to your architecture as it is ours, you have to build in rollback and safety, and you have to explicitly exercise those procedures to help your migration teams gain the required confidence when you have to adapt and respond to unforeseen situations.

If the Kafka knowledge of your organization lives in one or two people’s heads, Kafka migration projects are a great opportunity to start to build those foundations of learning that need to scale to others. Your teams need the agency and the confidence that wisdom, knowledge, and experience will give them. Find and build on those sources if you need to undertake something like this.

Success is not guaranteed, and your circumstances and experiences will vary from ours. But one thing I can say is that we can still do difficult things, and you can too. Kafka can be intimidating to dig into because it often serves as the beating heart of critical data streams. But if you are thoughtful about your migration designs, and your team is eager to learn and participate, you can bring them along and show them that they can do this.