Kafka Fundamentals and Architecture

What is Kafka Really?

Here's the thing everyone gets wrong at first: Kafka is not a message queue.

I know, I know - it looks like a queue. You put messages in, you take messages out. But that mental model will hurt you later.

Kafka is a distributed append-only log. Think of it like an ever-growing spreadsheet where:

• New rows are always added at the bottom (append-only)
• Old rows never change (immutable)
• Multiple people can read any row they want, anytime (replayable)
• The spreadsheet is split across multiple files (distributed)

Core Data Structure: Topics and Partitions

Imagine you're organizing a library:

• Topic = Subject category (e.g., "user-events", "payment-transactions")
• Partition = Physical shelf within that category
• Message = Individual book on a shelf
• Offset = Page number that never changes

Topic: user-clicks
├── Partition 0: [msg-0, msg-1, msg-2, msg-3, ...]
├── Partition 1: [msg-0, msg-1, msg-2, ...]
└── Partition 2: [msg-0, msg-1, msg-2, ...]

Each partition is an independent, ordered sequence. Message 0 in Partition 0 has nothing to do with Message 0 in Partition 1.

Key insight: Ordering is only guaranteed within a partition, not across partitions.

How Kafka Clusters Work

A Kafka cluster is a group of servers (called brokers) working together. Think of it like a restaurant:

• Broker = Chef in the kitchen
• Topic = Type of cuisine
• Partition = Specific dish
• Leader = Head chef for that dish
• Replica = Backup chef who knows the recipe

Replication Protocol

Every partition has:

• One leader broker (handles all reads/writes)
• Multiple replica brokers (backup copies for fault tolerance)

ISR (In-Sync Replicas) = The replicas that are "caught up" with the leader

min.insync.replicas = 2

ZooKeeper vs KRaft

Old way (ZooKeeper):

• Separate system managing cluster metadata
• Extra operational complexity
• Single point of failure

New way (KRaft - Kafka Raft):

• Kafka manages itself using Raft consensus
• Simpler deployment
• Better scalability
• Available since Kafka 3.3+

Recommendation: If you're starting fresh in 2025, use KRaft. ZooKeeper is being phased out.

When to Use Kafka

Key Takeaways

1. Kafka is a distributed commit log, not a queue
2. Partitions are your concurrency dial - more partitions = more parallelism
3. Ordering is only guaranteed within a partition
4. Use KRaft mode for new deployments
5. Choose Kafka for event streaming, not simple messaging