Share
Embed
Scale, in and out, with Kstreams and k8s
🎞video: https://youtu.be/t6Av8wfoFCw
👨🏾💻code: https://github.com/DivLoic/xke-kingof-scaling
Loïc DIVAD
February 12, 2019
Transcript
54 slides
-
LE MOIS DU CLOUD Scale in / scale out with
Kafka-Streams and Kubernetes Road to autoscaling - Loïc DIVAD 2 -
LE MOIS DU CLOUD 3 @LoicMDivad Hello! Developer @XebiaFr, …
(also #Data Engineer, #Spark Trainer, @DataXDay #Organiser, #Writer @blog.xebia.fr, #DataLover) I am Loïc DIVAD -
7 Fermeture du CFP : 31 Mars Edition 2019 du
DataXDay : 27 Juin Fin des early birds : 8 Avril dataxday.fr -
LE MOIS DU CLOUD 8 @LoicMDivad Hello! Developer @XebiaFr, …
(also #Data Engineer, #Spark Trainer, @DataXDay #Organiser, #Writer @blog.xebia.fr, #DataLover) I am Loïc DIVAD -
APP { "value":[ "c3ff8ab19d00d9e5", "e3ff8c72b600d9e5" ] } [ { "impact":0,
"key":"X", "direction":"DOWN", "type":"Missed", "level":"Pro", "game":"Neowave" }, ...] 9 -
10 APP APP APP { "value":[ "c3ff8ab19d00d9e5", "e3ff8c72b600d9e5" ] }
[ { "impact":0, "key":"X", "direction":"DOWN", "type":"Missed", "level":"Pro", "game":"Neowave" }, ...] 10 -
APP APP APP APP APP { "value":[ "c3ff8ab19d00d9e5", "e3ff8c72b600d9e5" ]
} [ { "impact":0, "key":"X", "direction":"DOWN", "type":"Missed", "level":"Pro", "game":"Neowave" }, ...] 11 -
LE MOIS DU CLOUD APP 15 Kafka clients and the
consumer polling system -
LE MOIS DU CLOUD APP 16 Kafka clients and the
consumer polling system -
LE MOIS DU CLOUD APP 17 Kafka clients and the
consumer polling system -
LE MOIS DU CLOUD APP 18 Kafka clients and the
consumer polling system -
LE MOIS DU CLOUD APP 19 Kafka clients and the
consumer polling system -
LE MOIS DU CLOUD topic-partition-0 topic-partition-1 topic-partition-2 topic-partition-N APP ◇
Every topic in Kafka is split into one or more partitions ◇ All the streaming tasks are executed through one or multiple threads of the same instance 21 Kafka-Streams and the consumer protocol -
LE MOIS DU CLOUD topic-partition-0 topic-partition-1 topic-partition-2 topic-partition-N APP APP
◇ Consumers from the same consumer group cooperate to consume data from topics. ◇ Every instance by joining the group triggers a partition rebalance. 22 Kafka-Streams and the consumer protocol -
LE MOIS DU CLOUD Kafka-Streams and the consumer protocol topic-partition-0
topic-partition-1 topic-partition-2 topic-partition-N APP APP APP APP ◇ The maximum parallelism is determined by the number of partitions of the input topic(s) 23 -
LE MOIS DU CLOUD Container Orchestration: K8s or the state
of the art ➔ Source: Kubernetes.io Documentation 25 -
LE MOIS DU CLOUD K8s: Support for custom metrics kind:
Deployment # deployment.yaml #... template: containers: - name: streaming-app # ... - name: prometheus-to-sd # ... adapter.yaml - name: custom-metrics-sd-adapter Your Streaming App Prometheus to Stackdriver https://gcr.io/google-containers/prometheus-to-sd Metrics Server https://gcr.io/google-containers/custom-metrics-stackdriver-adapter JMX metrics in a Prometheus format Stackdriver 27 -
LE MOIS DU CLOUD Stackdriver, for a full observability of
your app Monitoring and management for services, containers, applications, and infrastructure • It’s Modular (Monitoring, Logging, Trace … ) • It’s Full-stack insights • It’s Multi-cloud • … and has a rich set of integration tools 28 -
LE MOIS DU CLOUD K8s: Support for custom metrics kind:
Deployment # deployment.yaml #... template: containers: - name: streaming-app # ... - name: prometheus-to-sd # ... adapter.yaml - name: custom-metrics-sd-adapter Your Streaming App Prometheus to Stackdriver https://gcr.io/google-containers/prometheus-to-sd Metrics Server https://gcr.io/google-containers/custom-metrics-stackdriver-adapter JMX metrics in a Prometheus format Stackdriver 29 -
LE MOIS DU CLOUD K8s: Support for custom metrics #
jmx-exporter.conf --- global: scrape_interval: 1s evaluation_interval: 1s rules: - pattern: "kafka.consumer<type=..., topic=GAME-FRAME-RS, partition=(.*)><>(.*):(.*)" labels: { partition: $2, topic: GAME-FRAME-RS, metric: $3 } name: "consumer_lag_rs" type: GAUGE - pattern: "kafka.consumer<type=..., topic=GAME-FRAME-RQ, partition=(.*)><>(.*):(.*)" labels: { partition: $2, topic: GAME-FRAME-RQ, metric: $3 } name: "consumer_lag_rq" type: GAUGE 30 -
LE MOIS DU CLOUD K8s: Support for custom metrics $
curl -s localhost:9001 |grep -E --color=always \ "metric|topic|partition" consumer_lag_rq{client="...",metric="records-lag",partition="0",topic="GAME-FRAME-RQ",} 314562.0 consumer_lag_rq{client="...",metric="records-lag",partition="1",topic="GAME-FRAME-RQ",} 291183.0 consumer_lag_rq{client="...",metric="records-lag",partition="2",topic="GAME-FRAME-RQ",} 302551.0 consumer_lag_rq{client="...",metric="records-lag",partition="3",topic="GAME-FRAME-RQ",} 7319.0 consumer_lag_rq{client="...",metric="records-lag",partition="4",topic="GAME-FRAME-RQ",} 19329.0 consumer_lag_rq{client="...",metric="records-lag",partition="5",topic="GAME-FRAME-RQ",} 30698.0 consumer_lag_rs{client="...",metric="records-lag",partition="0",topic="GAME-FRAME-RS",} 312124.0 consumer_lag_rs{client="...",metric="records-lag",partition="1",topic="GAME-FRAME-RS",} 293837.0 consumer_lag_rs{client="...",metric="records-lag",partition="2",topic="GAME-FRAME-RS",} 302940.0 consumer_lag_rs{client="...",metric="records-lag",partition="3",topic="GAME-FRAME-RS",} 6594.0 consumer_lag_rs{client="...",metric="records-lag",partition="4",topic="GAME-FRAME-RS",} 15777.0 consumer_lag_rs{client="...",metric="records-lag",partition="5",topic="GAME-FRAME-RS",} 24881.0 31 -
LE MOIS DU CLOUD K8s: Support for custom metrics 1
With a build tool, create a Docker image of your streaming application lolo ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date 32 -
LE MOIS DU CLOUD K8s: Support for custom metrics lolo
➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary 2 Ensure you have a kubernetes cluster ready! Use a infra as code software… like terraform 33 -
LE MOIS DU CLOUD K8s: Support for custom metrics lolo
➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml 3 With kubectl create your configurations, secrets, pods and services 34 -
LE MOIS DU CLOUD K8s: Support for custom metrics lolo
➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 4 Enjoy your deployment! 35 -
LE MOIS DU CLOUD K8s: Support for custom metrics lolo
➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 36 -
LE MOIS DU CLOUD K8s: Support for custom metrics lolo
➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 37 -
LE MOIS DU CLOUD K8s: Support for custom metrics lolo
➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 38 -
LE MOIS DU CLOUD K8s: Support for custom metrics kind:
Deployment # deployment.yaml #... template: containers: - name: streaming-app # ... - name: prometheus-to-sd # ... adapter.yaml - name: custom-metrics-sd-adapter Your Streaming App Prometheus to Stackdriver https://gcr.io/google-containers/prometheus-to-sd Metrics Server https://gcr.io/google-containers/custom-metrics-stackdriver-adapter Stackdriver 39 -
LE MOIS DU CLOUD K8s: Horizontal Pod Autoscaler 41 ➔
Source: Kubernetes.io Documentation -
LE MOIS DU CLOUD K8s: Horizontal Pod Autoscaler 42 -
Kubernetes Resource - Periodically adjusts the number of replicas - Base on CPU usage in autoscaling/v1 - Memory and custom metrics are covered by the autoscaling/v2beta1 - Use the metrics.k8s.io API through a metric server ➔ Source: Kubernetes.io Documentation -
LE MOIS DU CLOUD Scale In / scale out with
kafka-streams and k8s 43 -
LE MOIS DU CLOUD Scale In / scale out with
kafka-streams and k8s 45 -
46 CONCLUSION States migration, changelog compaction, topology upgrades and k8s
StateFull Sets adoption are the next challenges to ease auto-scaling BUILD THE FUTURE 1. Kafka-Streams exposes relevant metrics related to stream processing 2. Consumer-lag is one of the key metrics to monitor in real time application 3. The cloud trends brings a set of powerful tools on which the Kafka community keep a close look -
LE MOIS DU CLOUD 51 The Horizontal Pod Autoscaler algorithm
depends on the current metric value and replica number desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )] ➢ A ratio of two will double the number of intances within the respect of maxReplicas ➢ By using targetAverageValue, the metric is computed by taking the average of the given metric across all Pods The number of replicas may fluctuating frequently due to the dynamic nature of the metrics, it’s called trashing ➢ --horizontal-pod-autoscaler-downscale-delay (default 5m0s) ➢ --horizontal-pod-autoscaler-upscale-delay (default 3m0s) Note: Both Kafka-Streams topology modification and HPA makes rolling update imposible -
54 Links and references https://www.confluent.io/kafka-summit-sf18/deploying- kafka-streams-applications https://www.youtube.com/watch?v=9cyXXmRlGWQ https://www.confluent.io/blog/apache-kafka-kubernetes -could-you-should-you https://kubernetes.io/blog/2018/04/13/local-persistent-v
olumes-beta/ https://stackoverflow.com/questions/49482873/how-to -deploy-kafka-stream-applications-on-kubernetes https://www.youtube.com/watch?v=9TOoThIKafo&list=P LhMG-8t0efEvJM5Bt2_zNLNVCEWcfRN0i https://www.confluent.io/blog/streaming-in-the-clouds- where-to-start Ideas from The event oriented architecture Concepts from Streams and tables white paper All the code on github DivLoic/xke-kingof-scaling Pictures: ◇ Stormtroopers Photo by Corey Motta on Unsplash ◇ Sreens from https://kubernetes.io/docs/tasks/ ◇ Coin-op machine by xoxo from the Noun Project ◇ Arcade game by Icons Producer from the Noun Project ◇ Mainframe by monkik from the Noun Project