Kafka Internals & Production Operations
Kafka is a distributed, replicated, append-only commit log, and almost every claim it makes about durability and ordering reduces to "a partition is a log, replicated to a few brokers, with one leader." This note builds from that primitive up through replication and ISR, exactly-once across topics, KRaft, and the production failures that page on-call at 3am: under-replicated partitions, unclean leader election, rebalancing storms, and consumer-lag avalanches.
Prerequisites: Logs, replication, consumer offsets, and basic distributed-systems failure modes.
Cover these firstQueues & Async MessagingReplication & Partitioning
After this: Reason about Kafka ordering, durability, consumer groups, rebalancing, and exactly-once boundaries.
Suggested first pass: Read sections 1–5, answer each section in your own words, then use the remaining failure modes and exercises as the advanced pass.
Content reviewed 21 June 2026 · References: Apache Kafka design documentation · Kafka consumer configuration
1. A topic is split into partitions. Each partition is an ordered, append-only log replicated to a few brokers, one of which is the leader.
2. Producers append to the leader; the leader replicates to followers; a write is "committed" once the in-sync replicas (ISR) have it. Consumers read committed records and track their own offset.
3. Ordering is per-partition only; parallelism is one consumer per partition per group. Most Kafka questions reduce to those two facts.
Kafka's storage model is a replicated, durable, append-only commit log. A consumer group can provide queue-like work sharing, but records are not deleted when one consumer reads them; they age out by retention or compaction. This explains replay and multiple independent consumer groups without pretending Kafka has only one usage model.
Everything difficult then hangs off two invariants. (1) The partition is the unit of ordering, replication, and parallelism: order holds within a partition, never across; throughput scales by adding partitions; a group cannot have more working consumers than partitions. (2) The leader plus ISR is the unit of durability: a record is safe once the in-sync replicas hold it, and every durability or availability knob (acks, min.insync.replicas, unclean election) is tuning what "in-sync" and "committed" mean. Those two invariants frame any Kafka design or incident.
The three structural concepts everything else builds on. Once these are clear, the rest of Kafka follows as corollaries.
| Concept | What it is | Why it exists |
|---|---|---|
| Partition | An ordered append-only log; a topic = N partitions | Unit of ordering, parallelism & scale |
| Replica | A copy of a partition on another broker (RF copies total) | Durability & availability on broker loss |
| Leader | The one replica that serves all reads/writes for a partition | Single writer → consistent ordering |
| Follower | Replicas that pull from the leader to stay current | Standbys ready to take over |
| ISR | In-Sync Replicas: followers caught up within replica.lag.time.max.ms | Defines who can be promoted & what "committed" means |
A record is committed (visible to consumers, safe) once every replica currently in the ISR has it. Not all replicas, just the in-sync set. A follower that falls behind (slow disk, GC, network) is removed from the ISR so it cannot hold up commits; when it catches up it rejoins. The ISR shrinks and grows dynamically, so one slow follower degrades durability margin but not latency, and only ISR members are eligible to become leader — they are the only replicas guaranteed to hold every committed record and can safely take over without rolling back any committed write. acks=all with min.insync.replicas=2 means "do not ack unless at least 2 in-sync copies hold it," the standard durability contract.
Partitions set the ceiling: maximum consumer parallelism in a group equals partition count, and the partition key decides ordering (same key, same partition, ordered). Partitions can be added, but with a caveat. Existing records stay put; it is future records for a key that may hash to a new partition, so per-key ordering breaks across the change point (records before and after the resize can be split). Removing partitions is also difficult. Size for peak future parallelism up front. Over-partitioning has its own cost: more files, more leader elections, longer rebalances. The number of partitions is a capacity-planning question, not a default.
Kafka's throughput comes from (1) sequential disk I/O (appending to a log is the disk's best case, not random seeks), (2) the OS page cache (recent log segments live in RAM that the kernel manages; Kafka deliberately keeps data out of the JVM heap and lets the OS cache it, so reads of fresh data never touch disk), and (3) zero-copy via sendfile, where the broker streams bytes straight from the page cache to the network socket without copying them through userspace or the JVM. One caveat applies: TLS/SSL defeats zero-copy. Encrypting the stream forces the data back through userspace to be encrypted, so sendfile is lost and CPU cost rises. Enabling encryption trades raw throughput for that protection.
Kafka guarantees order only within a partition. If all events for one entity must stay ordered (an account's transactions), they need a shared partition key (account_id) so they land on the same partition. Spreading them for throughput sacrifices ordering. A low-cardinality or skewed key (everyone's country) creates a hot partition: one broker saturates while others idle. The key choice controls ordering and balance at once, so it warrants first-class design attention.
When a leader broker dies, one of its followers must take over. Which replicas are eligible and how much data may be lost is the central durability-versus-availability dial.
- 1Leader for partition P dies. The controller (KRaft quorum, or old ZK controller) detects it via session/heartbeat loss.
- 2Controller picks a new leader from the ISR, a replica known to hold all committed records.
- 3Metadata update propagates; producers and consumers learn the new leader and redirect (they refresh metadata on
NOT_LEADER). - ✓No data loss. The new leader held every committed record, by definition of ISR.
- ⚠If the ISR is empty (all in-sync replicas also down), there is a choice: wait, or perform an unclean election (below).
acks=all writes start failing (safe, not a bug). Kill the leader → a replica is promoted from the ISR with no loss. With unclean election ON, killing until the ISR is empty promotes a lagging replica → silent data loss. Click a dead replica to revive it.acks(producer):0fire-and-forget (fastest, lossy),1leader-only ack (loses data if leader dies before replicating),allwait for full ISR (durable).min.insync.replicas(topic/broker): minimum ISR size for anacks=allwrite to succeed. With RF=3, set this to 2 to tolerate one broker down while still requiring two durable copies.- The combination that matters:
acks=allplusmin.insync.replicas=2plusRF=3gives the standard "no committed-data loss on single broker failure" contract. If the ISR drops below 2, producers getNotEnoughReplicasand the write fails rather than silently risking loss. That is correct behavior, not a bug.
If every ISR replica is down and only an out-of-sync follower remains, Kafka has two options. unclean.leader.election.enable=false (default, correct for payments): the partition goes offline and unavailable until an ISR member returns, preserving every committed record at the cost of availability. =true: promote the stale follower so the partition stays available, but it is missing committed records, so data is silently lost and offsets may even go backwards. Financial data keeps it false and accepts downtime; lossy telemetry may allow it. The choice is an explicit trade-off between the two.
Unlike majority-quorum systems, Kafka can commit with any ISR size ≥ min.insync.replicas and promote any ISR member; it does not need a majority. This lets it tolerate more failures for a given RF (RF=3 survives 2 followers lagging, as long as the leader and enough ISR remain) and keeps the hot path fast. The cost is that the ISR mechanism itself must be managed by a consistent controller (ZooKeeper historically, now KRaft), which is quorum-based. Kafka pushes consensus into the control plane and keeps the data plane leader-driven.
An append-only log cannot grow forever. Kafka has two distinct cleanup policies, and confusing them is a common mistake.
| Policy | Deletes by | Keeps | Use for |
|---|---|---|---|
| Retention (delete) | Age (retention.ms) or size (retention.bytes) | Everything within the window, then drops whole segments | Event streams, logs, metrics; time-bounded history |
| Compaction | Superseded keys | The latest value per key, forever | Changelogs, state, "current value of X" topics |
The default keeps records for retention.ms (e.g. 7 days) or until the partition hits retention.bytes, then deletes the oldest log segments wholesale. Deletion is by segment file (cheap, just an unlink), not per-record. This is what enables replay within the window and multiple consumer groups reading the same history. A larger window buys more replay ability at the cost of more disk, a straight cost-versus-recoverability trade.
Tiered storage addresses the case of long retention without large local disks. It offloads older, closed log segments to cheap object storage (S3/GCS) while keeping the hot tail on broker disk; consumers reading old offsets are served transparently from the remote tier. This breaks the old coupling where retention was bounded by the broker's local disk, so weeks or months can be kept for replay and reprocessing cheaply, and it makes brokers faster to rebalance and recover (less local data to move). The trade-offs: reads from the cold tier have higher latency, and it adds an object-store dependency. It is mainstream now rather than exotic, and applies to production designs with long retention or cross-region reprocessing.
For a topic where each key represents an entity's current state ("user 42's profile is X"), aging out old keys is undesirable; the goal is to keep the most recent record for every key indefinitely while garbage-collecting superseded versions. A background cleaner rewrites segments, dropping older records that share a key with a newer one. A record with a null value is a tombstone: it marks the key deleted and is itself eventually removed. This is how Kafka topics can act as a durable, replayable key-value changelog, which is what Kafka Streams' state stores and __consumer_offsets rely on.
Two pitfalls. (1) Compaction is background and lazy. The "active" segment is never compacted, and there is a configurable lag (min.cleanable.dirty.ratio, min.compaction.lag.ms), so a consumer reading the tail will still see duplicate keys and even un-GC'd tombstones for a while. A topic does not necessarily hold exactly one record per key at read time. (2) Policies can be combined (cleanup.policy=compact,delete) to compact and age out, but plain "compact" keeps keys forever, so a topic with unbounded distinct keys will grow without bound. Match the policy to whether the keyspace is bounded.
A consumer group splits a topic's partitions across its members. Each partition is owned by exactly one consumer in the group, giving horizontal scale with preserved per-partition order. Rebalancing is how the group reassigns partitions when membership changes, and it is the source of the most common operational pain.
- The group coordinator (a broker) assigns partitions to members; each partition goes to exactly one consumer. More consumers than partitions leaves the extras idle.
- Consumers track progress via committed offsets (stored in the compacted
__consumer_offsetstopic). On restart, they resume from the last commit. - Liveness comes from heartbeats (
session.timeout.ms) and progress frommax.poll.interval.ms; a consumer that misses either is considered dead and removed.
When the offset is committed decides the delivery guarantee, and it mirrors the producer durability behavior. Auto-commit (enable.auto.commit=true, every ~5s) is convenient but unsafe: it can commit offsets for records that were polled but not yet fully processed, so a crash loses them (at-most-once, silent data loss). Manual commit gives a choice. Committing after processing yields at-least-once (a crash after processing but before commit redelivers, so handlers must be idempotent); committing before processing yields at-most-once. The safe default for anything important is manual commit, process-then-commit, with idempotent handling of redelivery. For true exactly-once across the consume-to-produce loop, the offset commit goes inside the transaction, alongside the output produce, not as a separate commitSync — only then is consume-transform-produce atomic.
With the eager assignor, any membership change makes every consumer revoke all its partitions, then the group reassigns from scratch, a stop-the-world pause where nobody consumes. If consumers are flapping (slow processing blowing max.poll.interval.ms, a deploy rolling pods, a GC pause missing a heartbeat), each flap triggers another full rebalance, and the group spends more time rebalancing than consuming. This is a rebalance storm, and lag climbs the whole time. The usual root causes are processing slower than the poll interval, aggressive autoscaling, or undersized session timeouts.
Two improvements, usually combined as the CooperativeStickyAssignor. Sticky: on rebalance, keep each consumer's existing partitions where possible and only move the minimum needed, avoiding needless drop-and-refetch of partitions (and, for stateful apps, rebuilding local state). Cooperative (incremental): instead of revoking everything at once, do it in steps. Only the partitions that must move are revoked, and the rest keep consuming throughout, so there is no global stop-the-world. Together they turn a disruptive full reshuffle into a minimal, non-blocking adjustment.
Two more levers. Static membership (group.instance.id): a consumer that restarts within session.timeout.ms keeps its identity and its partitions, so a rolling deploy or quick crash-restart does not trigger a rebalance at all. The timeouts also matter: raise max.poll.interval.ms above worst-case processing time so slow batches do not get the consumer evicted. The goal is to make membership stable so rebalances are rare; sticky and cooperative assignors make the unavoidable rebalances cheap.
Kafka's EOS makes the consume → process → produce loop atomic inside Kafka, so a crash-and-retry never double-writes downstream topics or double-counts an input. It is built from two pieces: the idempotent producer and transactions.
enable.idempotence=true: the broker assigns the producer a PID and tracks a per-partition sequence number. If a producer retries a send (network blip, no ack received), the broker recognizes the duplicate sequence number and drops it, so retries do not create duplicate records. This alone turns at-least-once produce into exactly-once produce to a single partition, and it is cheap enough to be the default in modern Kafka.
One caveat: ordering-on-retry only holds if max.in.flight.requests.per.connection ≤ 5. The broker only dedups and reorders within that window, so a higher value can let a retried batch land out of order. Modern Kafka enforces ≤5 when idempotence is on. The "idempotent producer" guarantee depends on this limit.
A transactional.id producer can write to multiple partitions/topics AND commit its input offsets as a single atomic unit. Either everything commits or everything aborts. The consumer's offset commit goes into the same transaction as the output records, so "read input X, produced outputs Y, Z" is all-or-nothing. There is no window where the outputs were produced but the input was not recorded as consumed, or vice versa.
- 1initTransactions() registers the
transactional.idand fences any older producer with the same id (zombie fencing). - 2beginTransaction(), read input records, process them.
- 3send() output records to topic(s); sendOffsetsToTransaction() for the consumed input offsets, both inside the txn.
- ↻commitTransaction(): the broker writes a COMMIT marker; outputs and the offset commit become visible atomically.
- ✓Consumers with
isolation.level=read_committedonly ever see committed records; aborted txns are invisible. Effectively-once across the Kafka hop.
EOS is exactly-once inside Kafka only: producer → broker → broker → consumer. The moment a consumer writes to an external database, calls a payment API, or sends an email, it is back to at-least-once unless that step is also idempotent. The combination that works end-to-end is Kafka EOS for the messaging path plus idempotent external writes (upsert by key, or a provider idempotency key). This is the same "effectively-once = at-least-once delivery + idempotent handler" framing from Idempotency & Exactly-Once Payment Processing, and the saga/outbox story in Two-Phase Commit Protocol. True exactly-once delivery over a network is impossible (Two Generals); EOS is exactly-once processing within Kafka's boundary.
Transactions add coordination overhead (a transaction coordinator, commit markers in the log) and modestly cut throughput, and read_committed consumers can only read up to the Last Stable Offset: they must wait for in-flight transactions to commit or abort, which adds latency. EOS fits when duplicates are truly unacceptable (financial event processing); for metrics and logs, at-least-once plus idempotent sinks is simpler and faster. Switching everything to EOS reflexively is unwarranted.
The single most important Kafka health metric. A partition is under-replicated when its ISR is smaller than its replication factor, because one or more followers fell behind or their broker is down. No data is lost yet, but the durability margin is gone: if RF=3 and ISR drops to 2, one more failure reaches min.insync.replicas; drop to 1 and an acks=all producer starts failing writes. Causes include a dead or slow broker, network saturation, a follower with slow disk, or replication throttling during a reassignment. Alert on UnderReplicatedPartitions > 0; it is the canary for almost every Kafka incident.
When all ISR replicas for a partition are down and only an out-of-sync replica survives, enabling unclean election promotes that stale replica to keep the partition available, at the cost of losing every committed record the stale replica was missing; offsets can appear to move backward and confuse consumers. Keep unclean.leader.election.enable=false for anything correctness-critical (the partition goes offline instead, which is the safe failure), and treat "partition offline, no ISR" as a page, not an auto-heal. Allow it only for explicitly lossy topics.
Lag = (log end offset − committed offset): how far behind a group is. It becomes an avalanche when consumers cannot keep up with producers and the gap grows unboundedly. The failure is self-reinforcing: lag triggers slow processing → missed max.poll.interval.ms → consumer evicted → rebalance → even less consuming → more lag. If lag exceeds the retention window, the oldest unconsumed records are deleted before they are read, permanent data loss for that group. Fixes: scale consumers up to the partition count (no further help beyond that), raise partition count for more parallelism, speed up processing (batch, async sinks), and shed or parallelize work within a consumer. Monitor the lag trend, not just the absolute value; lag rising at a steady rate is the early warning.
Because a partition is strictly ordered, one un-processable record at the consumer's position blocks everything behind it in that partition; retrying forever stalls the partition and lag climbs. Defenses: bounded retries then route to a dead-letter topic and advance, validate and deserialize defensively, and decide deliberately whether skipping (with a DLQ) is preferable to halting given the ordering needs. (Same shape as the saga poison-message problem.)
This rebalance storm operationally presents as "lag rising while consumers look healthy." Root-cause it by checking rebalance frequency: flapping membership from slow processing, rolling deploys, or autoscaling. Fixes: CooperativeStickyAssignor, static membership, and raising max.poll.interval.ms above worst-case processing. A group rebalancing every few seconds is consuming almost nothing.
A low-cardinality or skewed partition key (everyone's country, or one celebrity account_id) routes a disproportionate share to one partition, so one broker and one consumer saturate while the rest idle, and that consumer's lag grows alone. Vnode-style tricks do not apply here; fixes are choosing a higher-cardinality key, adding a salt or sub-key for the hot entity, or handling the hot key specially. The partition key controls both ordering and balance, and skew breaks balance.
- You need a durable, replayable event log that multiple independent consumers read at their own pace.
- You want to decouple producers from consumers and absorb bursts (a slow consumer never blocks the producer).
- High throughput with ordered-per-key delivery (event sourcing, CDC pipelines, stream processing, metrics).
- You need the log as a backbone, fanning out to analytics, search, caches, and services from one source of truth.
Alternatives, chosen by actual need:
| System | Model | Best for | Why not Kafka |
|---|---|---|---|
| Kafka | Replayable partitioned log | High-throughput streams, replay, fan-out | — |
| RabbitMQ | Smart broker, queues + routing | Complex routing, per-message TTL, RPC | Lower throughput; not a replayable log |
| SQS / Pub-Sub | Managed queue | Simple decoupling, no ops, autoscale | Limited ordering/replay; vendor-managed |
| Pulsar | Log + tiered storage, multi-tenant | Geo, many tenants, queue+stream in one | More moving parts; smaller ecosystem |
| Kinesis | Managed log (AWS) | AWS-native streaming, no ops | AWS lock-in; shard limits/cost |
| Plain DB / outbox | Table as queue | Low volume, transactional with your data | Doesn't scale to stream throughput |
Kafka is a log, not a general-purpose queue. For per-message TTLs, priority queues, complex routing topologies, or low-volume RPC-style messaging, RabbitMQ or a managed queue is simpler and a better fit; bolting those onto Kafka fights the grain. Kafka is strongest when the replayable, ordered, high-throughput log is the actual requirement: event sourcing, CDC, stream processing, and fan-out to many consumers. The deciding factor is what the system is for, not its popularity.
The modern-operations cluster: how Kafka coordinates itself now, how a deployment spans data centers, and how the streams are processed.
Historically Kafka stored cluster metadata (brokers, topics, partition leaders, and ISR) in ZooKeeper. KRaft replaces that dependency with an internal Raft quorum of controller nodes backed by a metadata log. This simplifies operations and improves metadata recovery and scalability. Current Kafka releases use KRaft; capacity and controller-failover targets still need to be measured for the cluster's metadata size, controller quorum, and network.
Kafka clusters are single-DC for latency, so spanning regions uses asynchronous replication. MirrorMaker 2 (built on Kafka Connect) copies topics between clusters and also replicates consumer offsets and topic configs, so a consumer can fail over to the other DC and resume near where it left off. Two common topologies exist: active-passive (DR, one live and one standby) and active-active (both live, mirrored both ways, with topic prefixing like us-east.orders to prevent infinite replication loops). The catch is that replication is async, so a regional failover can lose the last unreplicated records (non-zero RPO) and exact offsets do not translate perfectly across clusters; design consumers to tolerate some replay or loss at failover.
Both do stateful stream processing (joins, windows, aggregations) with exactly-once. Kafka Streams is a library embedded in a service: no separate cluster, scaling by running more instances, with state in local RocksDB backed by compacted changelog topics. It fits Kafka-to-Kafka processing where operational simplicity matters. Flink is a distributed processing cluster with its own job/resource manager, richer windowing and event-time/watermark semantics, true checkpointed state with savepoints, and many sources and sinks beyond Kafka. Flink fits large, complex, multi-source pipelines with demanding event-time correctness and big state; Kafka Streams fits simpler, embedded, Kafka-centric transforms without standing up another cluster.
| Dimension | Kafka Streams | Flink |
|---|---|---|
| Deploy | Library in your app | Separate cluster + job/resource managers |
| State | Local RocksDB + changelog topic | Checkpointed distributed state + savepoints |
| Sources | Kafka in → Kafka out | Many sources/sinks (Kafka, files, DBs…) |
| Event-time | Supported, simpler | Rich watermarks, late-data handling |
| Best for | Embedded, Kafka-centric, ops-light | Large, complex, multi-source, big state |
The questions that follow once the happy path is drawn. Click each to expand the answer.
A record is committed once every replica currently in the ISR has it, not all RF replicas, just the in-sync set. The producer's acks decides what it waits for: acks=1 waits only for the leader (so if the leader dies before a follower copies the record, it is lost), and acks=all waits for the full ISR. min.insync.replicas is the floor on how big that ISR must be for an acks=all write to be accepted.
The standard durable config is RF=3, acks=all, min.insync.replicas=2: a write needs two in-sync copies to be acked, so losing one broker still means zero committed-data loss. If the ISR shrinks to 1, producers get NotEnoughReplicas and writes fail rather than silently risking loss, which is the correct, safe behavior. RF sets how many copies exist; min.insync sets how many must confirm; acks=all ties the producer to that contract.
Only when availability strictly beats durability for that topic: lossy telemetry, logs, or metrics where a gap is acceptable but downtime is not. With it enabled, if all ISR replicas are down and only a stale, out-of-sync replica remains, Kafka promotes that stale replica so the partition keeps serving. But it is missing committed records, so data is silently lost and offsets can even move backward, which confuses consumers.
For anything correctness-critical, such as payments or financial events, I keep it false (the default). In that same scenario the partition then goes offline until an ISR member comes back: downtime in exchange for guaranteeing no committed record is lost. The decision is per-topic and it is a deliberate durability-versus-availability trade, so I would state which I am choosing and why rather than leave it on an unexplained default.
First, is it a throughput problem or a rebalance problem? Check rebalance frequency: if the group rebalances every few seconds, it is barely consuming, and the root cause is flapping membership (processing slower than max.poll.interval.ms, rolling deploys, autoscaling). Fix with CooperativeStickyAssignor, static membership, and raising the poll interval above worst-case processing. If membership is stable but lag still grows, it is genuine under-capacity.
For under-capacity: scale consumers up to the partition count (beyond that, extra consumers idle, so raising the partition count may be needed for more parallelism), speed up per-record processing (batch writes, async/bulk sinks, parallelize within the consumer), and check for a hot partition skewing load onto one consumer. The urgent risk is that if lag exceeds the retention window, unconsumed records are deleted before they are read, which is permanent loss. So I would alert on lag trend and on lag approaching retention, and if needed temporarily raise retention to buy time while scaling.
With the eager assignor, any membership change makes every consumer revoke all its partitions and the group reassigns from scratch, a stop-the-world pause where nobody consumes. If consumers keep flapping (slow processing missing the poll interval, pods rolling, GC pauses missing heartbeats), each flap triggers another full rebalance and the group spends more time rebalancing than working. That is the storm, and lag climbs throughout.
The CooperativeStickyAssignor fixes both halves. Sticky: keep each consumer's current partitions where possible and move only the minimum, avoiding needless drop-and-refetch (or rebuilding local state for stateful apps). Cooperative/incremental: revoke partitions in steps rather than all at once, so only the partitions that must move pause and everything else keeps consuming, with no global stop-the-world. I would pair that with static membership so a quick restart does not rebalance at all, and tune timeouts so slow batches do not get consumers evicted. Sticky and cooperative assignors make unavoidable rebalances cheap; stable membership makes them rare.
Two layers. The idempotent producer gives each producer a PID and per-partition sequence numbers, so the broker drops duplicate retries, giving exactly-once produce to a partition. Transactions go further: a transactional.id producer writes to multiple partitions and commits the consumed input offsets in one atomic unit, so "read input, produce outputs, record progress" is all-or-nothing. Consumers set isolation.level=read_committed to ignore aborted transactions. That makes the consume→process→produce loop effectively-once within Kafka.
The guarantee stops at Kafka's boundary. The instant the consumer writes to an external DB, calls a payment API, or sends an email, it is back to at-least-once unless that step is independently idempotent. So end-to-end I combine Kafka EOS for the messaging path with idempotent external writes (upsert by key, or the provider's idempotency key). To be precise: true exactly-once delivery over a network is impossible (Two Generals); what ships is exactly-once processing = at-least-once delivery + idempotent handlers.
Retention (delete) keeps a time/size window and drops the oldest segments; it fits event streams, logs, and metrics that need bounded history and replay within the window. Compaction keeps the latest value per key forever (older versions of a key are GC'd, a null value is a tombstone that deletes the key); it fits changelogs and "current state of X" topics, like Kafka Streams' state stores or __consumer_offsets.
The decision hinges on whether the topic represents events (time-ordered facts, retention) or state (latest-per-key, compaction). Two pitfalls: compaction is lazy and background, so a tail reader still sees duplicate keys for a while, and one-record-per-key should not be assumed at read time; and plain compaction keeps keys forever, so an unbounded keyspace grows without bound (combine compact,delete if keys must also age out).
KRaft moves cluster metadata (brokers, topics, partition leaders, ISR) out of ZooKeeper and into an internal Raft quorum that stores it in a Kafka log. Operationally that is one system instead of two: no separate ZK ensemble to provision, secure, and monitor. Architecturally it removes a scaling bottleneck: metadata changes used to funnel through a ZK-backed controller, which capped how many partitions a cluster could manage and made controller failover slow because the new controller had to reload state from ZooKeeper.
With KRaft the controllers form a Raft quorum and rebuild in-memory metadata from a replicated log. This removes a separate ZooKeeper deployment and improves recovery and partition scalability, but neither a sub-second failover nor a particular partition count is universal. Validate both with the supported Kafka version and the intended metadata workload.
A Kafka cluster is single-DC for latency, so I span regions with asynchronous replication via MirrorMaker 2, which copies topics and also replicates consumer offsets and configs so a consumer can fail over and resume near where it left off. Topology is either active-passive (one live, one warm standby for DR) or active-active (both live, mirrored both ways with topic prefixes like us-east.orders to avoid infinite replication loops).
The catch is that replication is asynchronous: at a regional failover the last records that had not yet replicated are lost (non-zero RPO), and offsets do not translate perfectly across clusters, so consumers may replay or skip a little. So I design consumers to tolerate some duplication or loss at failover (idempotent processing again), set expectations on RPO with the business, and for truly zero-loss cross-region I would note that this is a much harder, latency-costly problem (synchronous stretch clusters) rather than imply MM2 provides it.
Partition count is the ceiling on consumer parallelism in a group (more consumers than partitions just idle), so I start from target throughput: estimate per-partition throughput (bounded by one consumer's processing rate and one broker's write rate) and divide desired total throughput by it, then add headroom for growth since shrinking is hard. I also factor in that the partition key must spread load without breaking required ordering.
But maximizing it is not the goal, because over-partitioning has real costs: more open file handles and memory per broker, longer leader elections and rebalances, more end-to-end latency, and slower recovery. Adding partitions later remaps keys (breaking per-key ordering), so I would rather size generously up front than reshard. Rule of thumb: enough partitions to hit peak parallelism with some headroom, not orders of magnitude more. It is a capacity-planning decision, and I would state the throughput math behind the number.
That is a poison pill, and it is bad precisely because a partition is strictly ordered: the consumer is stuck at that offset, so everything behind it in the partition is blocked and lag climbs while the consumer retries forever. Retrying harder will not help if the record itself is the problem (bad schema, un-deserializable, a downstream that always rejects it).
Fix: bound the retries, then route the bad record to a dead-letter topic and commit past it so the partition flows again, and alert on DLQ depth so it gets triaged. Handlers should validate and deserialize defensively and not crash on one bad message. The deliberate decision is ordering: skipping leaves a gap in the ordered stream, so for strict-order flows I confirm that DLQ-and-continue is acceptable (it usually is, since one bad record should not freeze a partition) rather than halting and paging a human. It is the same poison-message pattern as in the saga note, just at the Kafka consumer layer.
Create a Kafka topic with 6 partitions and a consumer group with 3 consumers (2 partitions each). Produce 10,000 messages keyed by user_id drawn from a skewed distribution (80% of messages from 5 user IDs). Use kafka-consumer-groups.sh --describe to observe per-partition lag and count. You should see 1–2 partitions with the majority of messages while others are nearly empty.
Then produce the same 10k messages with null keys (round-robin) and compare the distribution. The contrast shows why key choice is the primary source of partition imbalance in production. This is partition and replica behavior made observable — the skew you see in the lag dashboard is the same skew that causes one consumer to fall behind while others are idle.
Design a Kafka topic for a payments event stream. Walk through partition count selection: how many consumers will you run, must payments from the same account be processed in order (what key do you use), and what happens if one account generates 30% of all traffic (hot partition)? Configure acks, min.insync.replicas, and replication.factor for a system where losing a payment event is unacceptable.
Then: a broker in the ISR goes down while a producer is mid-batch. Trace what happens with acks=all — does the produce succeed, block, or fail? What does unclean.leader.election=true mean for the payments data on that partition, and why would you never enable it here? Then: a consumer crashes mid-processing at offset 450 with auto-commit enabled. How many messages get reprocessed on restart, and how do you make the consumer idempotent to handle it? Covers partitions and replicas, leader election and durability, consumer groups and rebalancing, and exactly-once semantics.
Your payments consumer group has 6 consumers on a 6-partition topic. Lag is growing at 50k msg/min. Walk through your options: adding more consumers (what triggers a rebalance, what's the stop-the-world impact, and when does adding consumers stop helping), increasing parallelism within a partition (is it possible while preserving per-account order, and how), and tuning fetch size and processing batch size.
Then: someone proposes enabling exactly-once semantics (EOS) to prevent duplicate payments during rebalance. Walk through what EOS actually costs: the producer idempotency overhead, the transactional coordinator round-trip, and the end-to-end latency increase. At 50k msg/min lag, does the EOS overhead make the problem better or worse? When would you prefer at-least-once with idempotent consumers (dedup by payment ID in the DB) over true EOS? Covers consumer groups and rebalancing, exactly-once semantics, and failure modes at production depth.