Install Kafka Cluster (3 Node) In Docker Compose with UI (Ubuntu)

3 min readAug 1, 2023

Introduce

Kafka is a powerful and widely adopted distributed streaming platform that has revolutionized the way organizations handle real-time data processing. Developed by LinkedIn and later open-sourced by the Apache Software Foundation, Kafka is designed to handle high-throughput, fault-tolerant, and scalable data streams. In this article, we will delve into the core concepts and architecture of Kafka, explore its key features, and understand how it has become a fundamental component of modern data-driven applications.

Core Concepts:

Topics: In Kafka, data is organized into topics, which represent individual streams of records. A topic can be considered as a category or feed where data producers publish messages, and data consumers subscribe to these topics to process the messages in real-time.
Partitions: Each topic is divided into one or more partitions to achieve horizontal scalability. Partitions allow Kafka to distribute the data across multiple servers, enabling parallel processing and load balancing. Each partition is an ordered and immutable sequence of records.
Producers: Producers are applications responsible for publishing data to Kafka topics. They write data records to specific partitions within topics based on the partitioning strategy defined.
Consumers: Consumers are applications that subscribe to one or more Kafka topics and process the data published by producers. Kafka supports both traditional “pull-based” and newer “push-based” consumer models.
Brokers: Kafka is implemented as a distributed system, and the individual nodes in the cluster are called brokers. Brokers are responsible for handling data replication, storage, and serving consumer requests.

2. Architecture:

Kafka’s architecture is based on a distributed publish-subscribe model. Producers send data to Kafka brokers, and consumers read data from brokers. The key components in the Kafka architecture include:

ZooKeeper: Kafka relies on Apache ZooKeeper to manage the distributed coordination of brokers and maintain metadata about topics, partitions, and consumer groups.
Brokers: Kafka brokers form the core of the system. They store and manage data across partitions, handle replication, and facilitate communication with producers and consumers.
Consumer Groups: Consumers can be organized into consumer groups, where each group shares the load of processing data from a topic. This allows parallel processing and enables scaling of consumer applications.

3. Key Features:

Scalability: Kafka’s distributed nature allows it to scale horizontally across multiple servers, accommodating increasing data volumes without sacrificing performance.
Fault Tolerance: By replicating data across multiple brokers, Kafka ensures fault tolerance. If a broker fails, another broker within the cluster can take over its responsibilities seamlessly.
Durability: Kafka persists data for a configurable period, providing data durability and allowing consumers to replay historical data as needed.
Low Latency: Kafka’s design prioritizes low-latency data streaming, making it ideal for real-time applications and event-driven architectures.

How To Install On Docker Compose

Requirements

Docker + Docker Compose
Spec Requirements can check this article https://docs.confluent.io/platform/current/installation/system-requirements.html

Step

Install Zookeeper and kafka nodes (3 node)

version: '2.1'

services:
  zoo1:
    image: confluentinc/cp-zookeeper:7.3.2
    hostname: zoo1
    container_name: zoo1
    ports:
      - "2181:2181"
    environment:
      ZOOKEEPER_CLIENT_PORT: 2181
      ZOOKEEPER_SERVER_ID: 1
      ZOOKEEPER_SERVERS: zoo1:2888:3888


  kafka1:
    image: confluentinc/cp-kafka:7.3.2
    hostname: kafka1
    container_name: kafka1
    ports:
      - "9092:9092"
      - "29092:29092"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka1:19092,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9092,DOCKER://host.docker.internal:29092
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 1
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
    depends_on:
      - zoo1

  kafka2:
    image: confluentinc/cp-kafka:7.3.2
    hostname: kafka2
    container_name: kafka2
    ports:
      - "9093:9093"
      - "29093:29093"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka2:19093,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9093,DOCKER://host.docker.internal:29093
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 2
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
    depends_on:
      - zoo1


  kafka3:
    image: confluentinc/cp-kafka:7.3.2
    hostname: kafka3
    container_name: kafka3
    ports:
      - "9094:9094"
      - "29094:29094"
    environment:
      KAFKA_ADVERTISED_LISTENERS: INTERNAL://kafka3:19094,EXTERNAL://${DOCKER_HOST_IP:-127.0.0.1}:9094,DOCKER://host.docker.internal:29094
      KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT,DOCKER:PLAINTEXT
      KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
      KAFKA_ZOOKEEPER_CONNECT: "zoo1:2181"
      KAFKA_BROKER_ID: 3
      KAFKA_LOG4J_LOGGERS: "kafka.controller=INFO,kafka.producer.async.DefaultEventHandler=INFO,state.change.logger=INFO"
      KAFKA_AUTHORIZER_CLASS_NAME: kafka.security.authorizer.AclAuthorizer
      KAFKA_ALLOW_EVERYONE_IF_NO_ACL_FOUND: "true"
    depends_on:
      - zoo1

save this file to docker-compose.yml on your server directory
cd into your directory

docker compose up -d

waiting to show like

 ✔ Network kafka_default  Created                                                                                                                                                                                                        0.3s 
 ✔ Container zoo1         Started                                                                                                                                                                                                       12.0s 
 ✔ Container kafka1       Started                                                                                                                                                                                                        2.9s 
 ✔ Container kafka3       Started                                                                                                                                                                                                        3.4s 
 ✔ Container kafka2       Started

Installing UI for manage your kafka

run this command

docker run -it -d -p 8080:8080 -e DYNAMIC_CONFIG_ENABLED=true provectuslabs/kafka-ui

open on your browser http://<your-ip>:8080/
Click configure new cluster
add your cluster like this

click Submit

Done

Install Kafka Cluster (3 Node) In Docker Compose with UI (Ubuntu)

Introduce

How To Install On Docker Compose

Written by Nasri Adzlani