28. Kafka를 이용한 데이터 동기화

1. 설치 : http://kafka.apache.org 

Apache Kafka

압축해제

%tar xvf kafka_2.13-3.3.1.tgz

2. kafka 3rd party library : https://cwiki.apache.org/confluence/display/KAFKA/Clients

Clients - Apache Kafka - Apache Software Foundation

3. zookeeper & kafka 서버실행

3-1. zookeeper 실행

$KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties

$KAFKA_HOME/bin/zookeeper-server-stop.sh

3-2. kafka 실행

$KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties

$KAFKA_HOME/bin/kafka-server-stop.sh

 

4. Topic

4-1. 생성

$KAFKA_HOME/bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092 --partitions 1

4-2. 목록

$KAFKA_HOME/bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

4-3. 상세

$KAFKA_HOME/bin/kafka-topics.sh  --describe --topic quickstart-events --bootstrap-server localhost:9092 

 

5. Message Produce & Subscribe

5-1. Producing

$KAFKA_HOME/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic quickstart-events

5-2. Subscribe

$KAFKA_HOME/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic quickstart-events --from-beginning

6. 구동화면

% ./bin/zookeeper-server-start.sh ./config/zookeeper.properties

zookeeper 실행

% ./bin/kafka-server-start.sh ./config/server.properties

kafka 실행

$./bin/kafka-topics.sh --bootstrap-server localhost:9092 --list

$./bin/kafka-topics.sh --create --topic quickstart-events --bootstrap-server localhost:9092 --partitions 1

토픽 생성 및 조회

[2022-12-12 00:29:54,136] WARN [AdminClient clientId=adminclient-1] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)

오류 발생시 아래 파일의 내용을 적용할 host로 교정한다

$vi config/server.properties

advertised.listeners=PLAINTEXT://your.host.name:9092

7. Kafka Producer & Consumer 기동

%./bin/kafka-console-producer.sh --broker-list localhost:9092 --topic quickstart-events

Producer

%./bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic quickstart-events --from-beginning

Consumer

Producer에서 발행을 하면 Consumer에서 동일하게 표현이 되고 있다

Consumer를 하나 더 기동하고 메시지를 발행하면 두 곳에서 메시지 확인가능