Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Running Apache Kafka on Kubernetes

Running Apache Kafka on Kubernetes

2020년 4월 25일, Global Azure Virtual 2020 에서 발표. (https://festa.io/events/976)

Presented at Global Azure Virtual 2020, April 25th, 2020.

* The slides have been slightly updated to adapt to the changes.
* For configuring Kafka cluster without helm chart with minimal effort, see: https://gist.github.com/dongjinleekr/fcadc20063553935cfb6536185421ca2

Slides: Korean. Presentation: Korean.

Lee Dongjin

April 25, 2020
Tweet

More Decks by Lee Dongjin

Other Decks in Technology

Transcript

  1. Running Apache Kafka on
    Kubernetes
    Lee Dongjin | [email protected]

    View Slide

  2. 오늘 할 이야기는...
    ● 오늘의 주제
    ○ Apache Kafka를 어떻게 Kubernetes + Docker 환경에서 띄울 것인가?
    ● 좀 더 구체적으로는...
    ○ 장점, 단점
    ○ 사용 방법

    View Slide

  3. ● 단점
    ○ Resource 문제
    ○ Monitoring 문제
    ○ Troubleshooting 문제
    ■ “뭔가 잘못됐는데, Kubernetes도 Kafka도 잘 아는 사람이 없어!”
    ● 장점
    ○ 쉽고 간편함
    ■ 방법만 안다면.
    ○ 표준화된 배포 환경
    ○ 빠른 전개 + Scale-up
    Kafka on Kubernetes: 장단점

    View Slide

  4. Kafka on Kubernetes: 왜 어렵지? (1)
    ● Zookeeper
    ○ zookeeper (Official)
    ○ wurstmeister/zookeeper
    ● Kafka
    ○ confluentinc/cp-kafka
    ○ wurstmeister/kafka
    ○ solsson/kafka
    ○ bitnami/kafka
    ○ spotify/kafka

    View Slide

  5. Kafka on Kubernetes: 왜 어렵지? (2)
    ● ‘상태가 있는' Application
    ○ StatefulSet
    ● Not-interchangeable
    ○ Broker A와 Broker B는 완전히 다르다.
    ■ 쥐고 있는 Topic
    ■ 쥐고 있는 Partition
    ■ 쥐고 있는 Partition의 내용: Leader vs. Follower
    ○ Client는 원하는 (Topic, Partition)의 Leader Partition을 쥔 Broker와만 직접 커뮤니케이션한다.
    ■ cf. MySQL

    View Slide

  6. Kafka on Kubernetes: 왜 어렵지? (3)
    ● Kafka 까지 설치했다고 끝이 아님
    ○ 이제 절반
    ○ 같은 kubernetes namespace 안에서 돌려야 하는 다른 요소들이 많음
    ● 설치 대상
    ○ 각종 테스트/운영용 Kafka client
    ■ 공식 client (kafka-topics.sh, ...)
    ■ kafkacat: 테스트 및 관리용 유틸리티
    ○ 기타 Ecosystem 툴
    ■ Schema Registry
    ■ Kafka Connect
    ■ ...

    View Slide

  7. Kafka on Kubernetes: 그러면 어떻게?
    ● Stateful Application (Zookeeper, Kafka)
    ○ Helm Chart (incubator)
    ■ incubator/zookeeper
    ■ incubator/kafka
    ○ 약간의 요령만 알면 쓸 수 있다.
    ● Stateless Application (clients, ecosystem tools)
    ○ 단순 Deploy

    View Slide

  8. Apache Zookeeper w/ Helm (1)
    ● Zookeeper (Based on Official Docker Image)
    ○ incubator/zookeeper
    ○ 이름하고 개수(default: 3) 정해 주면 OK!
    # Deploy Helm Chart w/ name = djlee-zookeeper
    $ helm install djlee-zookeeper incubator/zookeeper
    # ls
    $ kubectl -n default exec -it djlee-zookeeper-0 -- bin/zkCli.sh -server
    djlee-zookeeper-0.djlee-zookeeper-headless:2181,djlee-zookeeper-1.djlee-
    zookeeper-headless:2181 ls /

    View Slide

  9. Apache Zookeeper w/ Helm (2)
    # create: /foo => bar (from node 0)
    $ kubectl -n default exec -it djlee-zookeeper-0 -- bin/zkCli.sh -server
    djlee-zookeeper-0.djlee-zookeeper-headless:2181,djlee-zookeeper-1.djlee-
    zookeeper-headless:2181 create /foo bar
    # Success: exit w/ 0
    # get: /foo (from node 2)
    $ kubectl -n default exec -it djlee-zookeeper-2 -- bin/zkCli.sh -server
    djlee-zookeeper-0.djlee-zookeeper-headless:2181,djlee-zookeeper-1.djlee-
    zookeeper-headless:2181 get /foo
    # Success: returns ‘bar’

    View Slide

  10. Apache Kafka w/ Helm (1)
    ● Kafka (Based on solsson/kafka)
    ○ incubator/kafka
    ○ 이름, Zookeeper 설정, 사용할 Docker Image, Replica 개수 등
    ● Zookeeper 설정
    ○ zookeeper.enabled, zookeeper.url
    ○ default: "Broker 별로 Zookeeper Instance를 하나씩 띄운다."

    View Slide

  11. Apache Kafka w/ Helm (2)
    # broker.yaml
    replicas: 4 # Kafka Cluster의 크기 (3+1)
    image: "solsson/kafka"
    imageTag: "2.3.0" # 현재 default: 2.0.1
    zookeeper:
    enabled: false # 자동 Zookeeper 생성 기능을 사용하지 않음
    url:
    "djlee-zookeeper-0.djlee-zookeeper-headless:2181,djlee-zookeeper-1.djlee
    -zookeeper-headless:2181" # 아까 설치한 Zookeeper Cluster의 위치
    # Deploy Helm Chart w/ name = djlee-kafka
    $ helm install djlee-kafka -f broker.yaml incubator/kafka

    View Slide

  12. 테스트 툴 설치: Kafka 공식 clients (1)
    # clients.yaml
    apiVersion: v1
    kind: Pod
    metadata:
    name: clients
    namespace: default
    spec:
    containers:
    - name: clients
    image: confluentinc/cp-kafka:5.3.1 # Apache Kafka 2.3.1
    command:
    - sh
    - -c
    - "exec tail -f /dev/null"

    View Slide

  13. 테스트 툴 설치: Kafka 공식 clients (2)
    # Deploy
    $ kubectl apply -f clients.yaml
    # Topic 생성: test (partition = 3, replication factor = 3)
    $ kubectl -n default exec -it clients -- kafka-topics --zookeeper
    djlee-zookeeper-0.djlee-zookeeper-headless:2181,djlee-zookeeper-1.djlee-
    zookeeper-headless:2181 --topic test --create --partitions 3
    --replication-factor 3
    # Topic 안에 저장된 Records를 (맨 처음부터) 확인
    $ kubectl -n default exec -it clients -- kafka-console-consumer
    --bootstrap-server djlee-kafka-0.djlee-kafka-headless:9092 --topic test
    --from-beginning

    View Slide

  14. 테스트 툴 설치: Kafkacat (1)
    ● Kafkacat
    ○ edenhill/kafkacat
    ○ Non-JVM Kafka Producer / Consumer
    ○ netcat의 Kafka 판
    ● Produce, Consume, Metadata Listing 기능만 있음
    ○ 관리, 테스트용

    View Slide

  15. 테스트 툴 설치: Kafkacat (2)
    # kafkacat.yaml
    apiVersion: v1
    kind: Pod
    metadata:
    name: kafkacat
    namespace: default
    spec:
    containers:
    - name: kafkacat
    image: confluentinc/cp-kafkacat:latest
    command:
    - sh
    - -c
    - "exec tail -f /dev/null"

    View Slide

  16. 테스트 툴 설치: Kafkacat (3)
    # Deploy
    $ kubectl apply -f kafkacat.yaml
    # Topic Metadata 확인
    $ kubectl -n default exec -it kafkacat -- kafkacat \
    -b djlee-kafka-0.djlee-kafka-headless:9092 -L
    # Text File 내용을 ‘test’ Topic으로 Produce
    $ cat test.txt | kubectl -n default exec -it kafkacat -- kafkacat \
    -b djlee-kafka-0.djlee-kafka-headless:9092 -t test -P

    View Slide

  17. 정리
    ● Kafka on Kubernetes
    ○ 장점도, 단점도 있음.
    ○ 빠르고 간편하게 표준화된 배포 환경에 전개할 수 있다는 점은 확실한 장점.
    ● 사용법
    ○ Zookeeper w/ Helm
    ○ Kafka w/ Helm
    ○ etc (e.g. Kafka Clients): 단순 Deploy

    View Slide

  18. 질문?

    View Slide