⚙️ Kafka System — Interactive Explorer

Cluster Architecture

Click any component to inspect it.

Producers Brokers Topic / Partitions Consumer Groups Coordinator / Controller
Producer: Orders acks=all, idempotent Producer: Payments key=userId Producer: Web linger.ms=20 Kafka Cluster Broker 1 P0(L) P1(F) P2(F) Broker 2 P0(F) P1(L) P2(F) Broker 3 P0(F) P1(F) P2(L) Controller (KRaft) Metadata, leaders, ISR Group Coordinator Owns __consumer_offsets Topic: "orders" — 3 partitions, RF=3 P0 log P1 log P2 log Group: billing C1 ⇢ P0 C2 ⇢ P1 C3 ⇢ P2 Group: analytics C1 ⇢ P0,P1 C2 ⇢ P2 Stream Processor Kafka Streams / Flink Schema Registry Avro/Protobuf/JSON Kafka Connect Source / Sink

L = Leader replica, F = Follower replica. The leader handles all reads/writes for that partition.

Producer → Partition Log

Send messages and watch them get partitioned, appended to the log, and replicated.

Topic: orders — partitioning rule: hash(key) % numPartitions
How the message reaches the log
  1. Serializer turns key/value to bytes (optionally validated against Schema Registry).
  2. Partitioner picks a partition: murmur2(key) % N if key present; sticky/round-robin if not.
  3. Accumulator batches records per partition (controlled by linger.ms and batch.size).
  4. Sender ships batch to the partition leader broker.
  5. Leader appends to its on-disk segment, assigning the next offset.
  6. Followers fetch and replicate; once in-sync replicas (ISR) ack, the leader acks the producer (acks=all).

Consumer Groups & Partition Assignment

Add/remove consumers to see Kafka rebalance partitions across the group.

The rules of consumer groups
  • A consumer group is identified by group.id. Each partition is consumed by exactly one consumer inside the group.
  • If consumers > partitions, the extra consumers sit idle.
  • If consumers < partitions, some consumers handle multiple partitions.
  • Different groups read the same topic independently — each maintains its own offsets.
  • Joining/leaving triggers a rebalance coordinated by the Group Coordinator.

Offsets, Commits & Lag

Step the consumer forward through the log. Commit offsets to mark "processed".

Log End Offset (LEO): 0
Current Position: 0
Committed Offset: —
Consumer Lag: 0
Partition log (offsets are 0-indexed and append-only):
Stored message Last committed Current position
What "committing" actually does

The consumer sends a tiny record like {group: "billing", topic: "orders", partition: 0, offset: 17} to the internal compacted topic __consumer_offsets. On restart or rebalance, the new consumer for that partition resumes from the last committed offset + 1.

  • Auto-commit (enable.auto.commit=true): periodic, easy but can lose or duplicate messages.
  • Manual sync commit: consumer.commitSync() after successful processing → at-least-once.
  • Transactional: produce + commit-offset atomically → exactly-once within Kafka.

Replication & ISR

Toggle followers to see how the In-Sync Replica set protects your data.

How acks and ISR interact
  • acks=0: fire-and-forget. Fastest, can lose data.
  • acks=1: leader writes locally and acks. Loss possible if leader crashes before replication.
  • acks=all + min.insync.replicas=2: leader waits for all ISR followers to replicate. Strongest durability.
  • If a follower lags beyond replica.lag.time.max.ms, it's kicked out of ISR.
  • If leader dies, the controller elects a new leader from ISR.

End-to-End Message Lifecycle

Step through what happens to a single message, from producer.send() to commitSync().

Step 1 / 9