Scale, in and out, with Kstreams and k8s

🎞video: https://youtu.be/t6Av8wfoFCw
👨🏾‍💻code: https://github.com/DivLoic/xke-kingof-scaling

Transcript

1. Le mois du cloud 1
2. LE MOIS DU CLOUD Scale in / scale out with Kafka-Streams and Kubernetes Road to autoscaling - Loïc DIVAD 2
3. LE MOIS DU CLOUD 3 @LoicMDivad Hello! Developer @XebiaFr, … (also #Data Engineer, #Spark Trainer, @DataXDay #Organiser, #Writer @blog.xebia.fr, #DataLover) I am Loïc DIVAD
4. 4 contact@dataxday.fr June 27th, Paris
6. 6 dataxday.fr
7. 7 Fermeture du CFP : 31 Mars Edition 2019 du DataXDay : 27 Juin Fin des early birds : 8 Avril dataxday.fr
8. LE MOIS DU CLOUD 8 @LoicMDivad Hello! Developer @XebiaFr, … (also #Data Engineer, #Spark Trainer, @DataXDay #Organiser, #Writer @blog.xebia.fr, #DataLover) I am Loïc DIVAD
9. APP { "value":[ "c3ff8ab19d00d9e5", "e3ff8c72b600d9e5" ] } [ { "impact":0, "key":"X", "direction":"DOWN", "type":"Missed", "level":"Pro", "game":"Neowave" }, ...] 9
10. 10 APP APP APP { "value":[ "c3ff8ab19d00d9e5", "e3ff8c72b600d9e5" ] } [ { "impact":0, "key":"X", "direction":"DOWN", "type":"Missed", "level":"Pro", "game":"Neowave" }, ...] 10
11. APP APP APP APP APP { "value":[ "c3ff8ab19d00d9e5", "e3ff8c72b600d9e5" ] } [ { "impact":0, "key":"X", "direction":"DOWN", "type":"Missed", "level":"Pro", "game":"Neowave" }, ...] 11
12. LE MOIS DU CLOUD 12
13. LE MOIS DU CLOUD 13 Producers Consumers Kafka Broker-1 Broker-1 Broker-1 topic
14. LE MOIS DU CLOUD Streaming Apps Consumer polling 14
15. LE MOIS DU CLOUD APP 15 Kafka clients and the consumer polling system
16. LE MOIS DU CLOUD APP 16 Kafka clients and the consumer polling system
17. LE MOIS DU CLOUD APP 17 Kafka clients and the consumer polling system
18. LE MOIS DU CLOUD APP 18 Kafka clients and the consumer polling system
19. LE MOIS DU CLOUD APP 19 Kafka clients and the consumer polling system
20. LE MOIS DU CLOUD Streaming Apps Kafka-Streams and the consumer protocol 20
21. LE MOIS DU CLOUD topic-partition-0 topic-partition-1 topic-partition-2 topic-partition-N APP ◇ Every topic in Kafka is split into one or more partitions ◇ All the streaming tasks are executed through one or multiple threads of the same instance 21 Kafka-Streams and the consumer protocol
22. LE MOIS DU CLOUD topic-partition-0 topic-partition-1 topic-partition-2 topic-partition-N APP APP ◇ Consumers from the same consumer group cooperate to consume data from topics. ◇ Every instance by joining the group triggers a partition rebalance. 22 Kafka-Streams and the consumer protocol
23. LE MOIS DU CLOUD Kafka-Streams and the consumer protocol topic-partition-0 topic-partition-1 topic-partition-2 topic-partition-N APP APP APP APP ◇ The maximum parallelism is determined by the number of partitions of the input topic(s) 23
24. LE MOIS DU CLOUD Container Orchestration Kubernetes, or the state of the art 24
25. LE MOIS DU CLOUD Container Orchestration: K8s or the state of the art ➔ Source: Kubernetes.io Documentation 25
26. LE MOIS DU CLOUD Container Orchestration Support for custom metrics 26
27. LE MOIS DU CLOUD K8s: Support for custom metrics kind: Deployment # deployment.yaml #... template: containers: - name: streaming-app # ... - name: prometheus-to-sd # ... adapter.yaml - name: custom-metrics-sd-adapter Your Streaming App Prometheus to Stackdriver https://gcr.io/google-containers/prometheus-to-sd Metrics Server https://gcr.io/google-containers/custom-metrics-stackdriver-adapter JMX metrics in a Prometheus format Stackdriver 27
28. LE MOIS DU CLOUD Stackdriver, for a full observability of your app Monitoring and management for services, containers, applications, and infrastructure • It’s Modular (Monitoring, Logging, Trace … ) • It’s Full-stack insights • It’s Multi-cloud • … and has a rich set of integration tools 28
29. LE MOIS DU CLOUD K8s: Support for custom metrics kind: Deployment # deployment.yaml #... template: containers: - name: streaming-app # ... - name: prometheus-to-sd # ... adapter.yaml - name: custom-metrics-sd-adapter Your Streaming App Prometheus to Stackdriver https://gcr.io/google-containers/prometheus-to-sd Metrics Server https://gcr.io/google-containers/custom-metrics-stackdriver-adapter JMX metrics in a Prometheus format Stackdriver 29
30. LE MOIS DU CLOUD K8s: Support for custom metrics # jmx-exporter.conf --- global: scrape_interval: 1s evaluation_interval: 1s rules: - pattern: "kafka.consumer<type=..., topic=GAME-FRAME-RS, partition=(.*)><>(.*):(.*)" labels: { partition: $2, topic: GAME-FRAME-RS, metric: $3 } name: "consumer_lag_rs" type: GAUGE - pattern: "kafka.consumer<type=..., topic=GAME-FRAME-RQ, partition=(.*)><>(.*):(.*)" labels: { partition: $2, topic: GAME-FRAME-RQ, metric: $3 } name: "consumer_lag_rq" type: GAUGE 30
31. LE MOIS DU CLOUD K8s: Support for custom metrics $ curl -s localhost:9001 |grep -E --color=always \ "metric|topic|partition" consumer_lag_rq{client="...",metric="records-lag",partition="0",topic="GAME-FRAME-RQ",} 314562.0 consumer_lag_rq{client="...",metric="records-lag",partition="1",topic="GAME-FRAME-RQ",} 291183.0 consumer_lag_rq{client="...",metric="records-lag",partition="2",topic="GAME-FRAME-RQ",} 302551.0 consumer_lag_rq{client="...",metric="records-lag",partition="3",topic="GAME-FRAME-RQ",} 7319.0 consumer_lag_rq{client="...",metric="records-lag",partition="4",topic="GAME-FRAME-RQ",} 19329.0 consumer_lag_rq{client="...",metric="records-lag",partition="5",topic="GAME-FRAME-RQ",} 30698.0 consumer_lag_rs{client="...",metric="records-lag",partition="0",topic="GAME-FRAME-RS",} 312124.0 consumer_lag_rs{client="...",metric="records-lag",partition="1",topic="GAME-FRAME-RS",} 293837.0 consumer_lag_rs{client="...",metric="records-lag",partition="2",topic="GAME-FRAME-RS",} 302940.0 consumer_lag_rs{client="...",metric="records-lag",partition="3",topic="GAME-FRAME-RS",} 6594.0 consumer_lag_rs{client="...",metric="records-lag",partition="4",topic="GAME-FRAME-RS",} 15777.0 consumer_lag_rs{client="...",metric="records-lag",partition="5",topic="GAME-FRAME-RS",} 24881.0 31
32. LE MOIS DU CLOUD K8s: Support for custom metrics 1 With a build tool, create a Docker image of your streaming application lolo ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date 32
33. LE MOIS DU CLOUD K8s: Support for custom metrics lolo ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary 2 Ensure you have a kubernetes cluster ready! Use a infra as code software… like terraform 33
34. LE MOIS DU CLOUD K8s: Support for custom metrics lolo ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml 3 With kubectl create your configurations, secrets, pods and services 34
35. LE MOIS DU CLOUD K8s: Support for custom metrics lolo ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 4 Enjoy your deployment! 35
36. LE MOIS DU CLOUD K8s: Support for custom metrics lolo ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 36
37. LE MOIS DU CLOUD K8s: Support for custom metrics lolo ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 37
38. LE MOIS DU CLOUD K8s: Support for custom metrics lolo ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 38
39. LE MOIS DU CLOUD K8s: Support for custom metrics kind: Deployment # deployment.yaml #... template: containers: - name: streaming-app # ... - name: prometheus-to-sd # ... adapter.yaml - name: custom-metrics-sd-adapter Your Streaming App Prometheus to Stackdriver https://gcr.io/google-containers/prometheus-to-sd Metrics Server https://gcr.io/google-containers/custom-metrics-stackdriver-adapter Stackdriver 39
40. LE MOIS DU CLOUD Container Orchestration Horizontal Pod Autoscaler 40
41. LE MOIS DU CLOUD K8s: Horizontal Pod Autoscaler 41 ➔ Source: Kubernetes.io Documentation
42. LE MOIS DU CLOUD K8s: Horizontal Pod Autoscaler 42 - Kubernetes Resource - Periodically adjusts the number of replicas - Base on CPU usage in autoscaling/v1 - Memory and custom metrics are covered by the autoscaling/v2beta1 - Use the metrics.k8s.io API through a metric server ➔ Source: Kubernetes.io Documentation
43. LE MOIS DU CLOUD Scale In / scale out with kafka-streams and k8s 43
44. LE MOIS DU CLOUD 44
45. LE MOIS DU CLOUD Scale In / scale out with kafka-streams and k8s 45
46. 46 CONCLUSION States migration, changelog compaction, topology upgrades and k8s StateFull Sets adoption are the next challenges to ease auto-scaling BUILD THE FUTURE 1. Kafka-Streams exposes relevant metrics related to stream processing 2. Consumer-lag is one of the key metrics to monitor in real time application 3. The cloud trends brings a set of powerful tools on which the Kafka community keep a close look
47. LE MOIS DU CLOUD 47 MERCI
48. LE MOIS DU CLOUD 48
49. LE MOIS DU CLOUD 49 ANNEXES
50. LE MOIS DU CLOUD 50
51. LE MOIS DU CLOUD 51 The Horizontal Pod Autoscaler algorithm depends on the current metric value and replica number desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )] ➢ A ratio of two will double the number of intances within the respect of maxReplicas ➢ By using targetAverageValue, the metric is computed by taking the average of the given metric across all Pods The number of replicas may fluctuating frequently due to the dynamic nature of the metrics, it’s called trashing ➢ --horizontal-pod-autoscaler-downscale-delay (default 5m0s) ➢ --horizontal-pod-autoscaler-upscale-delay (default 3m0s) Note: Both Kafka-Streams topology modification and HPA makes rolling update imposible
52. LE MOIS DU CLOUD 52 now supports more than https://www.confluent.io/blog/apache-kafka-supports-200k-partitions-per-cluster 200K partitions
53. LE MOIS DU CLOUD 53
54. 54 Links and references https://www.confluent.io/kafka-summit-sf18/deploying- kafka-streams-applications https://www.youtube.com/watch?v=9cyXXmRlGWQ https://www.confluent.io/blog/apache-kafka-kubernetes -could-you-should-you https://kubernetes.io/blog/2018/04/13/local-persistent-v olumes-beta/ https://stackoverflow.com/questions/49482873/how-to -deploy-kafka-stream-applications-on-kubernetes https://www.youtube.com/watch?v=9TOoThIKafo&list=P LhMG-8t0efEvJM5Bt2_zNLNVCEWcfRN0i https://www.confluent.io/blog/streaming-in-the-clouds- where-to-start Ideas from The event oriented architecture Concepts from Streams and tables white paper All the code on github DivLoic/xke-kingof-scaling Pictures: ◇ Stormtroopers Photo by Corey Motta on Unsplash ◇ Sreens from https://kubernetes.io/docs/tasks/ ◇ Coin-op machine by xoxo from the Noun Project ◇ Arcade game by Icons Producer from the Noun Project ◇ Mainframe by monkik from the Noun Project