Split View: Apache Kafka 설치 guide (with Zookeeper)
Apache Kafka 설치 guide (with Zookeeper)
Overview
Apache Kafka 설치 방법에 대해서 알아본다.
Zookeeper 설치 방법
Kafka를 설치하기 위해서 Zookeeper가 필요하다. 최근에는 Zookeeper 대신 KRaft 라는 것을 사용할 수 있다고 한다. Zookeeper 가 생소한 분들을 위해 간략하게 소개하자면, 분산 어플리케이션을 위한 config 저장소라고 보면 된다. Hadoop의 특성상, 분산된 환경에서 application이 동작하게 되는데, 필연적으로 전역 정보를 저장하고 싶거나, 동기화를 위한 lock 같은 것이 필요할 때가 있다. 이 때, zookeeper를 사용하면된다. zookeeper 공식 홈페이지에서는 아래와 같이 설명하고 있다.
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
Zookeeper tar 다운로드 및 압축 해제
현재 시점(20221214) Zookeeper의 가장 최신 release 버젼은 3.8.0 이고, 아래 사이트에서 tar.gz을 다운로드 받을 수 있다. https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
tar -zxvf apache-zookeeper-3.8.0-bin.tar.gz
zoo.cfg 수정
Zookeeper 서버는 3대로 구성할 것이기 때문에 server.1,server.2,server.3 에 차례로 zookeeper 서버의 fqdn과 사용할 포트 범위 2888:3888을 합쳐 입력해준다. (fqdn 대신 IP 주소를 넣어도 무방하다.)
dataDir 즉 myid 파일을 저장할 위치를 수정한다. 필자는 /var/zookeeper 로 지정했다.
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/var/zookeeper
# the port at which the clients will connect
clientPort=2181
server.1=server1:2888:3888
server.2=server2:2888:3888
server.3=server3:2888:3888
myid 파일 생성
위에서 지정한 dataDir 에 서버마다 고유한 myid를 입력해 준다.
예를 들면, server1에는 아래와 같이 입력하고
1
server2에는 아래와 같이 입력한다.
2
1~255까지의 숫자만 입력가능하다.
zookeeper server 구동
Zookeeper가 설치된 서버에서 명령어를 실행하여, Zookeeper를 구동시킨다.
apache-zookeeper-3.8.0-bin/bin/zkServer.sh start
Kafka 설치
Kafka binary 다운로드 및 압축 해제
wget https://dlcdn.apache.org/kafka/3.3.1/kafka_2.13-3.3.1.tgz
tar -xzf kafka_2.13-3.3.1.tgz
server.properties 수정
Zookeeper 설치시 myid 파일을 서버별로 변경해줬던 것 처럼, kafka server별로 broker.id 를 고유하게 설정해준다.
zookeeper.connect 에는 Zookeeper가 Cluster 정보를 입력해준다.
broker.id=1
zookeeper.connect=server1:2181,server2:2181,server3:2181
kafka broker 실행
kafka_2.13-3.3.1/bin/kafka-server-start.sh config/server.properties
topic 생성
kafka를 실행했던 node 중 한 곳에서, 아래와 같은 명령어로 test라는 topic을 생성해보자. Created topic test. 라는 문구가 출력되면 정상적으로 설치된 것이다.
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 20 --topic test
Apache Kafka Installation Guide (with Zookeeper)
Overview
Learn how to install Apache Kafka.
Zookeeper Installation
Zookeeper is required to install Kafka. Recently, it has become possible to use KRaft instead of Zookeeper. For those unfamiliar with Zookeeper, it can be briefly described as a configuration store for distributed applications. Due to the nature of Hadoop, applications run in distributed environments, and there are inevitably times when you need to store global information or need locks for synchronization. In such cases, you can use Zookeeper. The official Zookeeper website describes it as follows:
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services.
Download and Extract Zookeeper Tar
At the current time (20221214), the latest release version of Zookeeper is 3.8.0, and the tar.gz can be downloaded from the site below. https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
wget https://dlcdn.apache.org/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
tar -zxvf apache-zookeeper-3.8.0-bin.tar.gz
Modify zoo.cfg
Since we will configure the Zookeeper server with 3 nodes, enter the FQDN of each Zookeeper server along with the port range 2888:3888 for server.1, server.2, and server.3 respectively. (You may use IP addresses instead of FQDNs.)
Modify dataDir, which is the location where the myid file will be stored. I set it to /var/zookeeper.
# The number of milliseconds of each tick
tickTime=2000
# The number of ticks that the initial
# synchronization phase can take
initLimit=10
# The number of ticks that can pass between
# sending a request and getting an acknowledgement
syncLimit=5
# the directory where the snapshot is stored.
# do not use /tmp for storage, /tmp here is just
# example sakes.
dataDir=/var/zookeeper
# the port at which the clients will connect
clientPort=2181
server.1=server1:2888:3888
server.2=server2:2888:3888
server.3=server3:2888:3888
Create myid File
Enter a unique myid for each server in the dataDir specified above.
For example, on server1 enter the following:
1
On server2, enter the following:
2
Only numbers between 1 and 255 are allowed.
Start Zookeeper Server
Run the command on the server where Zookeeper is installed to start Zookeeper.
apache-zookeeper-3.8.0-bin/bin/zkServer.sh start
Kafka Installation
Download and Extract Kafka Binary
wget https://dlcdn.apache.org/kafka/3.3.1/kafka_2.13-3.3.1.tgz
tar -xzf kafka_2.13-3.3.1.tgz
Modify server.properties
Just as you changed the myid file for each server during Zookeeper installation, set a unique broker.id for each Kafka server.
Enter the Zookeeper cluster information in zookeeper.connect.
broker.id=1
zookeeper.connect=server1:2181,server2:2181,server3:2181
Start Kafka Broker
kafka_2.13-3.3.1/bin/kafka-server-start.sh config/server.properties
Create a Topic
On one of the nodes where Kafka was started, create a topic called test using the command below. If the message Created topic test. is displayed, the installation was successful.
bin/kafka-topics.sh --create --bootstrap-server localhost:9092 --replication-factor 3 --partitions 20 --topic test
Quiz
Q1: What is the main topic covered in "Apache Kafka Installation Guide (with Zookeeper)"?
Learn how to install Apache Kafka (with Zookeeper).
Q2: What is Download and Extract Zookeeper Tar?
At the current time (20221214), the latest release version of Zookeeper is 3.8.0, and the tar.gz
can be downloaded from the site below.
https://www.apache.org/dyn/closer.lua/zookeeper/zookeeper-3.8.0/apache-zookeeper-3.8.0-bin.tar.gz
Q3: Explain the core concept of Modify zoo.cfg.
Since we will configure the Zookeeper server with 3 nodes, enter the FQDN of each Zookeeper server
along with the port range 2888:3888 for server.1, server.2, and server.3 respectively.
Q4: What are the key aspects of Create myid File?
Enter a unique myid for each server in the dataDir specified above. For example, on server1 enter
the following: On server2, enter the following: Only numbers between 1 and 255 are allowed.
Q5: How does Start Zookeeper Server work?
Run the command on the server where Zookeeper is installed to start Zookeeper. Kafka Installation