Overview

Learn how to install Apache Kafka.

Zookeeper Installation

Zookeeper is required to install Kafka. Recently, it has become possible to use KRaft instead of Zookeeper. For those unfamiliar with Zookeeper, it can be briefly described as a configuration store for distributed applications. Due to the nature of Hadoop, applications run in distributed environments, and there are inevitably times when you need to store global information or need locks for synchronization. In such cases, you can use Zookeeper. The official Zookeeper website describes it as follows:

ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.

Download and Extract Zookeeper Tar

At the current time (20221214), the latest release version of Zookeeper is 3.8.0, and the tar.gz can be downloaded from the site below. https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz

wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
tar -zxvf apache-zookeeper-3.8.0-bin.tar.gz

Modify zoo.cfg

Since we will configure the Zookeeper server with 3 nodes, enter the FQDN of each Zookeeper server along with the port range 2888:3888 for server.1, server.2, and server.3 respectively. (You may use IP addresses instead of FQDNs.) Modify dataDir, which is the location where the myid file will be stored. I set it to /var/zookeeper.

# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/var/zookeeper
# the port at which the clients will connect
clientPort=2181
server.1=server1:2888:3888
server.2=server2:2888:3888
server.3=server3:2888:3888

Create myid File

Enter a unique myid for each server in the dataDir specified above.

For example, on server1 enter the following:

On server2, enter the following:

Only numbers between 1 and 255 are allowed.

Start Zookeeper Server

Run the command on the server where Zookeeper is installed to start Zookeeper.

apache-zookeeper-3.8.0-bin/bin/zkServer.sh start

Kafka Installation

Download and Extract Kafka Binary

wget https://dlcdn.apache.org/kafka/3.3.1/kafka_2.13-3.3.1.tgz
tar -xzf kafka_2.13-3.3.1.tgz

Modify server.properties

Just as you changed the myid file for each server during Zookeeper installation, set a unique broker.id for each Kafka server. Enter the Zookeeper cluster information in zookeeper.connect.

broker.id=1
zookeeper.connect=server1:2181,server2:2181,server3:2181

Start Kafka Broker

kafka_2.13-3.3.1/bin/kafka-server-start.sh config/server.properties

Create a Topic

On one of the nodes where Kafka was started, create a topic called test using the command below. If the message Created topic test. is displayed, the installation was successful.

bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 20 --topic test

Zookeeper Installation
Kafka Installation
Create a Topic