Kafka Reliability Assumptions That Will Cost You Orders
Engineering notes from production experience with high-throughput event-driven systems.
Introduction: The Illusion of Fire-and-Forget
Kafka feels magical when you first use it. You produce a message, it disappears into the queue, and somewhere downstream a consumer handles it. No blocking. No tight coupling. Clean architecture diagrams with arrows going into a neat orange box labelled “Kafka.”
And then at 3 AM your phone rings, a large batch of orders has silently failed to process, and the Kafka consumer pod is sitting there — perfectly green, perfectly healthy — completely silently refusing to do anything.
This is not a theoretical concern. Across several years of running Kafka in a high-throughput order-processing system, a small set of reliability assumptions shows up again and again as the root cause behind major incidents — each one preventable, each one caused by an engineer assuming Kafka would handle something it explicitly does not.
This post is a walkthrough of those assumptions, with real (genericized) examples, code patterns, and fixes. The goal isn’t to scare you away from Kafka — it’s one of the best pieces of infrastructure you can build on. The goal is to make sure you respect it.
Part of a bigger list: this is a deep dive into one entry from 10 Production Failure Patterns I Keep Seeing — here we zoom all the way in on the Kafka one.
Assumption 1: “If the Pod is Running, the Consumer is Working”
This is the most dangerous assumption of all, because every standard monitoring setup makes it feel true. Your dashboard shows the pod as Running, CPU flat, memory stable — everything looks fine. Meanwhile, your consumer has been processing zero messages for hours.
In one incident, a consumer responsible for syncing order status between two internal systems was deployed with a message parsing bug. Instead of crashing, it just stopped consuming — no error, no restart, no alert. Kafka kept the messages waiting patiently in the queue, but the consumer never came back for them. Downstream systems fell behind, and by the time anyone noticed, thousands of orders had been affected.
The reason this is so easy to miss: Kafka consumers are pull-based. There’s no dropped connection or failed request to alert on — the consumer stays “healthy” in the healthcheck sense even while silently failing on every message. Unlike an HTTP server, where a bug shows up instantly as errors, a broken consumer only shows up in lag — a metric almost nobody watches on day one.
The Fix: Monitor Lag, Not Pods
Consumer health is not pod.status == Running. Consumer health is consumer lag approaching zero.
# Grafana alert — add this for every critical consumer
alert: KafkaConsumerLagHigh
expr: kafka_consumer_group_lag{group="order-status-consumer"} > 1000
for: 5m
labels:
severity: critical
annotations:
summary: "Consumer {{ $labels.group }} is lagging — possible silent failure"
The rule of thumb: if your consumer group’s lag is growing, treat it as an outage, not a warning. Alert at a lag threshold and page on-call. Don’t wait for someone to notice a downstream metric drop 45 minutes later.
The non-negotiable rule: Every Kafka consumer must have a consumer lag alert. Pod health alerts alone are necessary but not sufficient for consumer health.
Assumption 2: “acks=1 Means My Message is Safe”
This one is subtle. The Kafka acks setting controls how many broker replicas must acknowledge a produce request before the producer considers it “done.” Most engineers know acks=0 is dangerous (fire-and-forget, no confirmation at all) but assume acks=1 is fine — it means the partition leader confirmed receipt.
What they don’t account for is what happens when the partition leader isn’t there.
A Real Incident
An infrastructure team performed a routine managed-Kafka cluster version upgrade. Version upgrades trigger a rolling restart of brokers — one at a time, gracefully. Normally this is invisible to applications.
Except that several services were using acks=0 or acks=1 with an outdated version of a shared internal Kafka client library. When the rolling restart hit a particular broker, that broker’s partitions temporarily lost their leader — leader election takes a few hundred milliseconds.
During those milliseconds, producers using acks=1 sent messages to the old leader address. The library received no error — it just… received no acknowledgment. And in the old library version, “no acknowledgment within timeout” was treated as a silent no-op rather than a retryable error.
Messages were permanently lost. No error was raised anywhere. A couple hundred orders required manual dispatch and reconciliation.
The Acks Settings, Explained Simply
Think of it like this. You’re sending an important letter (a message).
acks=0— You drop it in the mailbox and walk away. You have no idea if anyone ever received it.acks=1— The postman at the first sorting office signs for it. But if he has a heart attack before passing it along, the letter is gone and nobody told you.acks=all— Every sorting office in the delivery chain signs for it. Only when all of them confirm do you get a receipt. This is durable.
// ❌ Don't do this for business-critical topics
cfg := sarama.NewConfig()
cfg.Producer.RequiredAcks = sarama.WaitForLocal // acks=1 — leader only
producer, _ := sarama.NewSyncProducer(brokers, cfg)
// ✅ Do this instead
cfg := sarama.NewConfig()
cfg.Producer.RequiredAcks = sarama.WaitForAll // acks=all — all ISR replicas
cfg.Producer.Retry.Max = 5
cfg.Producer.Return.Errors = true
producer, _ := sarama.NewSyncProducer(brokers, cfg)
The cost of acks=all is a small increase in produce latency (a few extra milliseconds for replica acknowledgment). The cost of acks=1 during a cluster maintenance window is silent, unrecoverable message loss.
The non-negotiable rule: All Kafka producers on business-critical topics must use
acks=all. There is no valid reason to useacks=0oracks=1when orders, payments, or inventory are involved.
Assumption 3: “If Produce Returned No Error, The Message Was Delivered”
Even if you use acks=all, there’s another silent failure mode that bites teams in production: what if the produce call is cancelled before it completes?
A Real Incident
A broker in a managed Kafka cluster entered a hardware healing state. Kafka brokers don’t disappear immediately in this scenario — they respond slowly, with increasing latency. The order registration flow worked roughly like this:
- User places an order →
POST /order/registeris called - The handler registers the order in the DB
- The handler publishes a message to Kafka to trigger downstream processing
- The handler returns a success response
The HTTP handler had a request-scoped context with a timeout of a few seconds. When the broker was degraded, the Kafka publish at step 3 started taking longer and longer, creeping toward that timeout.
At that point, the Go runtime cancelled the context — the request timeout fired. The Kafka client received a context.Canceled error. The produce call was abandoned. The order was already written to the DB (step 2 succeeded), but the downstream Kafka event was never sent.
Well over a hundred orders ended up permanently stuck in a “purchased but not progressing” state — the DB said they existed, but no downstream system ever received the signal to continue processing them. They were eventually cancelled by an ETA-breach cleanup job, which is a terrible user experience.
The Fix: Decouple Kafka Publish from Request Context
This is a subtle but critical design pattern. Your HTTP request context should gate the response to the user — but it should not gate your async event publishing, which needs to outlive the HTTP request regardless.
// ❌ Dangerous: Kafka publish dies when request times out
func (h *OrderHandler) RegisterOrder(ctx context.Context, req *OrderRequest) (*OrderResponse, error) {
order, err := h.db.CreateOrder(ctx, req)
if err != nil {
return nil, err
}
// If ctx is cancelled (timeout), this produce call is abandoned
// The order is in the DB but nobody downstream knows about it
err = h.producer.Publish(ctx, "order-events", order)
if err != nil {
return nil, err
}
return &OrderResponse{OrderID: order.ID}, nil
}
// ✅ Safe: Kafka publish uses a background context, independent of request lifecycle
func (h *OrderHandler) RegisterOrder(ctx context.Context, req *OrderRequest) (*OrderResponse, error) {
order, err := h.db.CreateOrder(ctx, req)
if err != nil {
return nil, err
}
// Background context — this publish will complete even if the HTTP request is done
publishCtx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
err = h.producer.Publish(publishCtx, "order-events", order)
if err != nil {
// Log and trigger reconciliation, don't fail the request
h.logger.Error("Kafka publish failed — triggering reconciliation", "orderID", order.ID, "err", err)
}
return &OrderResponse{OrderID: order.ID}, nil
}
The deeper lesson here is that any time you write to a database and then publish to Kafka, you have a two-phase commit problem. The DB write and the Kafka publish are two separate operations with no atomic guarantee between them. You must design for the case where one succeeds and the other fails.
The pattern to adopt: Kafka publish contexts must always be decoupled from request contexts. And every flow where a DB write precedes a Kafka publish must have a reconciliation mechanism to detect and recover orphaned records.
Assumption 4: “My Consumer Handles All Message Shapes”
Kafka topics are append-only logs. This means old messages sit in the topic forever (until retention kicks in), and new consumer code can encounter messages that were written by old producer code. This is the schema evolution problem, and it bites teams more often than they expect.
Real Incidents
One case: A team maintained two copies of a proto stub file in the same repository. When the consumer was updated to use the new stub, the PR accidentally left the old one in place. The consumer was resolving the wrong stub at build time. When deployed, the consumer silently parsed every message incorrectly — the deserialized struct had zero-value fields where data was expected. Roughly a thousand orders were dropped before the discrepancy was noticed.
Another case: A consumer was updated to use a new struct type for a cached object it read during processing. Old pods still running as part of a rolling deployment were consuming messages that referred to the new type, hitting nil pointer dereferences in goroutines with no recover(). The consumer crashed in a loop, and over a thousand orders were lost before it was caught.
The Underlying Problem
When you change what a consumer expects from a Kafka message — either by changing the proto/struct definition or by changing what it reads from side stores during processing — you create a compatibility window where old and new code coexist:
Timeline:
T0 ──────────────── T1 (deploy) ──── T2 (fully rolled) ──────────
old producers ← mixed zone → new consumers
writing msg v1 v1 and v2 msgs expect msg v2
on same topic!
During the rolling deployment between T1 and T2, both old and new messages can be in the queue simultaneously. A new consumer reading an old-format message must handle it gracefully — not panic.
// ❌ Fragile: assumes message is always the new format
func processMessage(msg []byte) error {
var event NewOrderEvent
if err := proto.Unmarshal(msg, &event); err != nil {
return err // consumer crashes, stops processing, lag builds up
}
return handleEvent(event)
}
// ✅ Resilient: handles unknown formats gracefully, routes to DLQ
func processMessage(msg []byte) error {
var event NewOrderEvent
if err := proto.Unmarshal(msg, &event); err != nil {
// Don't crash — route to DLQ for human review
log.Warn("Failed to parse message — routing to DLQ", "err", err)
return h.dlq.Publish(context.Background(), msg)
}
if event.OrderId == "" {
// Looks like an old-format message — handle with backward-compatible logic
return h.handleLegacyMessage(msg)
}
return handleEvent(event)
}
The rule: Consumers must never crash on unparseable messages. The choice is: skip (log + move on), retry (exponential backoff), or DLQ. Crashing and restarting is the worst option — you’ll just crash on the same message again.
Assumption 5: “At-Least-Once Delivery Is Good Enough”
Kafka’s standard delivery guarantee is at-least-once — every message will be delivered, but it might be delivered more than once (during retries, rebalances, or replays). Most teams accept this and say “we’ll handle duplicates.”
But handling duplicates is harder than it sounds, especially when your consumer’s processing involves writes to multiple systems.
The Real-World Consequence
Imagine a consumer that processes an ORDER_PLACED event and does two things:
- Inserts a row into an
orderstable - Calls an external payment service to initiate a charge
If this consumer crashes halfway through — after step 1, before step 2 — Kafka will redeliver the message. Your consumer will run again. Step 1 will fail with a unique constraint violation (row already exists). If you’re not handling this correctly, your code either silently skips the payment call (order never charged) or crashes and loops forever.
The fix is to design every consumer for idempotent processing:
func (c *OrderConsumer) processOrderPlaced(ctx context.Context, event *OrderPlacedEvent) error {
// Idempotent DB insert — use ON CONFLICT DO NOTHING or check existence first
order, err := c.db.UpsertOrder(ctx, event.OrderId, event.Details)
if err != nil {
return fmt.Errorf("upsert failed: %w", err)
}
// Check if payment was already initiated (idempotency key)
if order.PaymentInitiated {
log.Info("Duplicate message — payment already initiated, skipping", "orderId", event.OrderId)
return nil // Safe to commit offset
}
// Only call payment if we haven't already
return c.paymentService.InitiateCharge(ctx, order)
}
The Dead Letter Queue: Your Last Safety Net
Every assumption above has one common fix that reduces the blast radius of any consumer failure: the Dead Letter Queue (DLQ).
A DLQ is just another Kafka topic where you route messages that your consumer failed to process after N retries. Instead of crashing, skipping, or blocking the entire partition on a poison-pill message, you route it aside and keep moving.
flowchart LR
A[Kafka Topic] --> B[Consumer]
B -->|Success| C[Commit Offset]
B -->|Parse Error| D[DLQ Topic]
B -->|Transient Error| E[Retry with backoff]
E -->|Max retries exceeded| D
D --> F[Manual Review / Alert]
After one particularly painful incident, the action item was explicit: all critical Kafka topics must have a DLQ configured. This is a good engineering standard to adopt proactively rather than reactively. A consumer without a DLQ is a consumer that will either lose messages silently or stop processing entirely when it hits any unexpected message format.
Summary: The Rules
The condensed checklist, distilled from all of the above:
| # | Rule |
|---|---|
| 1 | Monitor consumer lag, not just pod health |
| 2 | Business-critical producers must use acks=all |
| 3 | Decouple the Kafka publish context from the request context |
| 4 | Consumers must handle unknown message formats gracefully — never crash |
| 5 | Design every consumer for idempotent processing |
| 6 | Give every critical topic a Dead Letter Queue |
Closing Thought
Kafka is not a fire-and-forget system. It’s a durable log — and that durability is only as reliable as the code you wrap around it. Every time an engineer says “it’ll probably be fine,” there’s a Kafka incident waiting to happen.
The good news is that none of these failures are exotic. They all come from the same place: trusting default settings, skipping testing for async components, and not instrumenting the right signals. Build the habits, run the checklists, add the lag alerts — and you’ll sleep a lot better.