KRaft Mode

Introduction

KRaft is Kafka's new consensus mechanism that removes the need for Zookeeper. This lesson covers how KRaft works, its benefits, and why it simplifies Kafka's architecture.

What is KRaft?

KRaft stands for Kafka Raft. It's a consensus algorithm built directly into Kafka, eliminating the need for Zookeeper in broker coordination.

Key Capabilities

• Leader Elections: Handles leadership decisions natively within Kafka
• Replication Management: Manages data replication between brokers
• Metadata Coordination: Stores and manages cluster metadata internally
• Simplified Architecture: Removes external dependency on Zookeeper

KRaft significantly simplifies Kafka's architecture by handling all coordination tasks internally.

How Does KRaft Work?

KRaft uses the Raft consensus algorithm to coordinate leader elections and replication directly between Kafka brokers. There's no need for an external system like Zookeeper.

The Coordination Process

Brokers communicate with each other directly:

• Each broker maintains awareness of cluster state
• When a broker goes down, KRaft ensures another broker takes over seamlessly
• The leader election process is streamlined and fully integrated into Kafka
• Metadata is replicated across controller nodes for redundancy

Failure Handling

When a broker fails:

1. Other brokers detect the failure
2. KRaft initiates leader election
3. A new leader is elected from available brokers
4. Clients are redirected to the new leader
5. Normal operation resumes

Benefits of KRaft

1. Self-Managed Metadata

Kafka can now manage its own metadata without needing a separate Zookeeper cluster. Everything is handled internally, making the system more efficient and reducing operational overhead.

2. Simplified Architecture

By eliminating Zookeeper, you remove an entire component from your infrastructure:

• No separate Zookeeper cluster to maintain
• Fewer moving parts to monitor
• Reduced complexity in deployment and operations

3. Improved Scalability

KRaft simplifies Kafka's scalability, making it easier to run large-scale deployments:

• Faster metadata operations
• Better performance at scale
• More efficient resource utilization

4. Reduced Operational Overhead

Without Zookeeper:

• Fewer services to manage and monitor
• Simpler configuration
• Lower infrastructure costs
• Easier troubleshooting

KRaft in Action: Broker Failure Example

Let's examine how KRaft handles failures in a cluster with three brokers (Broker1, Broker2, Broker3).

Normal Operation

All brokers send heartbeats to each other to indicate they're alive and functioning. Unlike Zookeeper mode, there's no central coordinator - brokers coordinate among themselves.

1
2Broker1 <---> Broker2
3Broker1 <---> Broker3
4Broker2 <---> Broker3

When Broker1 Fails

#### Step 1: Detecting the Failure

Broker1 stops responding. Broker2 and Broker3 detect the missed heartbeat and recognize that Broker1 is down.

#### Step 2: Initiating Leader Election

Both Broker2 and Broker3 check heartbeat status and initiate the leader election process among themselves.

#### Step 3: Electing New Leader

Through the Raft consensus algorithm, Broker2 is elected as the new leader.

#### Step 4: Client Redirection

When a client tries to connect to Broker1:

1. Client sends request to Broker1
2. Broker1 is unresponsive
3. Client requests metadata from other brokers
4. Client discovers Broker2 is the new leader
5. Client redirects requests to Broker2

   #### Step 5: Continued Operation

   The cluster remains operational with Broker2 handling requests, ensuring no disruption to service.

Sequence of Events

1. Missed Heartbeat: Broker1 stops sending heartbeats
2. Detection: Broker2 and Broker3 detect the failure
3. Election Start: Both brokers initiate leader election
4. Leader Ready: Broker2 becomes ready to lead
5. Leader Elected: Broker2 is officially elected
6. Client Request: Client attempts to reach Broker1
7. No Response: Broker1 doesn't respond
8. Metadata Request: Client asks for cluster metadata
9. Redirection: Broker2 provides updated information
10. New Request: Client sends requests to Broker2

    This process ensures the cluster stays operational without disruption, even when brokers fail.

KRaft vs Zookeeper

| Aspect | Zookeeper Mode | KRaft Mode |

|--------|---------------|------------|

| External Dependencies | Requires Zookeeper cluster | Self-contained |

| Architecture | Two separate systems | Single integrated system |

| Operational Complexity | Higher (two systems to manage) | Lower (one system) |

| Metadata Management | External (Zookeeper) | Internal (Kafka) |

| Scalability | Good | Better |

| Failover Speed | Fast | Faster |

Installing Kafka in KRaft Mode

Prerequisites

Before installing, ensure you have:

• Java Development Kit (JDK) 8 or higher (java -version to check)
• Platform-specific tools:

- Windows: Git Bash or PowerShell

- Mac: Homebrew package manager

- Linux: Default package manager (apt/yum)

• Docker: Docker and Docker Compose (for containerized setup)

Linux Installation

1. Download and Extract Kafka

   bash

   1   mkdir -p /opt/kafka
   2   wget https://downloads.apache.org/kafka/3.8.0/kafka_2.13-3.8.0.tgz
   3   tar -xvzf kafka_2.13-3.8.0.tgz -C /opt/kafka --strip-components=1

2. Create Log Directory

   bash

   1   mkdir /tmp/kraft-combined-logs

3. Configure server.properties

   bash

   1   cat <<EOL > /opt/kafka/config/kraft/server.properties
   2   process.roles=broker,controller
   3   node.id=1
   4   controller.quorum.voters=1@localhost:9093
   5   listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
   6   log.dirs=/tmp/kraft-combined-logs
   7   EOL

   Configuration explained:

   - process.roles=broker,controller: Node serves both roles

   - node.id=1: Unique node identifier

   - controller.quorum.voters: Defines controller quorum voters

   - listeners: Sets up client (9092) and controller (9093) listeners

   - log.dirs: Directory for storing Kafka logs

4. Format Storage

   bash

   1   /opt/kafka/bin/kafka-storage.sh format -t $(uuidgen) -c /opt/kafka/config/kraft/server.properties

5. Start Kafka

   bash

   1   /opt/kafka/bin/kafka-server-start.sh /opt/kafka/config/kraft/server.properties

   Mac Installation

6. Install Kafka via Homebrew

   bash

   1   brew install kafka

7. Create Log Directory

   bash

   1   mkdir /tmp/kraft-combined-logs

8. Configure server.properties

   bash

   1   cat <<EOL > /usr/local/etc/kafka/server.properties
   2   process.roles=broker,controller
   3   node.id=1
   4   controller.quorum.voters=1@localhost:9093
   5   listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
   6   log.dirs=/tmp/kraft-combined-logs
   7   EOL

9. Format Storage

   bash

   1   /usr/local/bin/kafka-storage format -t $(uuidgen) -c /usr/local/etc/kafka/server.properties

10. Start Kafka

    bash

    1   kafka-server-start /usr/local/etc/kafka/server.properties

    Windows Installation

11. Create Directory and Extract

    bash

    1   mkdir C:\kafka
    2   tar -xvzf kafka_2.13-3.8.0.tgz -C C:\kafka

12. Create Log Directory

    bash

    1   mkdir C:\tmp\kraft-combined-logs

13. Configure server.properties

    bash

    1   cat <<EOL > C:\kafka\config\kraft\server.properties
    2   process.roles=broker,controller
    3   node.id=1
    4   controller.quorum.voters=1@localhost:9093
    5   listeners=PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
    6   log.dirs=C:\tmp\kraft-combined-logs
    7   EOL

14. Format Storage

    bash

    1   C:\kafka\bin\windows\kafka-storage.bat format -t $([guid]::NewGuid()) -c C:\kafka\config\kraft\server.properties

15. Start Kafka

    bash

    1   C:\kafka\bin\windows\kafka-server-start.bat C:\kafka\config\kraft\server.properties

    Docker Installation

    #### Create docker-compose.yaml

    Create a directory for Kafka and prepare the Docker configuration:

    yaml

    1version: '3.8'
    2services:
    3  kafka-init:
    4    image: confluentinc/cp-kafka:7.5.0
    5    command: >
    6      bash -c "kafka-storage format -t $(kafka-storage random-uuid)
    7      -c /etc/kafka/server.properties && echo 'Storage formatted successfully'"
    8    volumes:
    9      - ./kafka-logs:/var/lib/kafka/data
    10    environment:
    11      KAFKA_NODE_ID: 1
    12      KAFKA_PROCESS_ROLES: broker,controller
    13      KAFKA_LISTENERS: PLAINTEXT://localhost:9092,CONTROLLER://localhost:9093
    14      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@localhost:9093
    15      KAFKA_LOG_DIRS: /var/lib/kafka/data
    16
    17  kafka:
    18    image: confluentinc/cp-kafka:7.5.0
    19    ports:
    20      - "9092:9092"
    21    volumes:
    22      - ./kafka-logs:/var/lib/kafka/data
    23    environment:
    24      KAFKA_NODE_ID: 1
    25      KAFKA_PROCESS_ROLES: broker,controller
    26      KAFKA_LISTENERS: PLAINTEXT://0.0.0.0:9092,CONTROLLER://0.0.0.0:9093
    27      KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://localhost:9092
    28      KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka:9093
    29      KAFKA_LOG_DIRS: /var/lib/kafka/data
    30    depends_on:
    31      - kafka-init

    #### Key Environment Variables

    • KAFKA_PROCESS_ROLES: Defines roles as broker and controller
    • KAFKA_CONTROLLER_QUORUM_VOTERS: Identifies nodes for controller quorum
    • KAFKA_LISTENERS: Sets up listeners for client and controller communications
    • KAFKA_LOG_DIRS: Points to log storage directory

    #### Start Services

    bash

    1docker-compose up -d

    The -d flag runs services in detached mode (background).

Summary

KRaft represents the future of Kafka architecture:

• Simpler deployment: No Zookeeper cluster needed
• Better performance: Native metadata management
• Easier operations: Single system to manage
• Improved scalability: Optimized for large-scale deployments
• Easy installation: Available on all major platforms

By embedding coordination directly into Kafka, KRaft reduces complexity while maintaining all the reliability and fault tolerance that Kafka is known for.