Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Scale, in and out, with Kstreams and k8s

Ed81876bf33da90cdae47ce9b8df056b?s=47 Loïc DIVAD
February 12, 2019

Scale, in and out, with Kstreams and k8s

Ed81876bf33da90cdae47ce9b8df056b?s=128

Loïc DIVAD

February 12, 2019
Tweet

Transcript

  1. Le mois du cloud 1

  2. LE MOIS DU CLOUD Scale in / scale out with

    Kafka-Streams and Kubernetes Road to autoscaling - Loïc DIVAD 2
  3. LE MOIS DU CLOUD 3 @LoicMDivad Hello! Developer @XebiaFr, …

    (also #Data Engineer, #Spark Trainer, @DataXDay #Organiser, #Writer @blog.xebia.fr, #DataLover) I am Loïc DIVAD
  4. 4 contact@dataxday.fr June 27th, Paris

  5. 5

  6. 6 dataxday.fr

  7. 7 Fermeture du CFP : 31 Mars Edition 2019 du

    DataXDay : 27 Juin Fin des early birds : 8 Avril dataxday.fr
  8. LE MOIS DU CLOUD 8 @LoicMDivad Hello! Developer @XebiaFr, …

    (also #Data Engineer, #Spark Trainer, @DataXDay #Organiser, #Writer @blog.xebia.fr, #DataLover) I am Loïc DIVAD
  9. APP { "value":[ "c3ff8ab19d00d9e5", "e3ff8c72b600d9e5" ] } [ { "impact":0,

    "key":"X", "direction":"DOWN", "type":"Missed", "level":"Pro", "game":"Neowave" }, ...] 9
  10. 10 APP APP APP { "value":[ "c3ff8ab19d00d9e5", "e3ff8c72b600d9e5" ] }

    [ { "impact":0, "key":"X", "direction":"DOWN", "type":"Missed", "level":"Pro", "game":"Neowave" }, ...] 10
  11. APP APP APP APP APP { "value":[ "c3ff8ab19d00d9e5", "e3ff8c72b600d9e5" ]

    } [ { "impact":0, "key":"X", "direction":"DOWN", "type":"Missed", "level":"Pro", "game":"Neowave" }, ...] 11
  12. LE MOIS DU CLOUD 12

  13. LE MOIS DU CLOUD 13 Producers Consumers Kafka Broker-1 Broker-1

    Broker-1 topic
  14. LE MOIS DU CLOUD Streaming Apps Consumer polling 14

  15. LE MOIS DU CLOUD APP 15 Kafka clients and the

    consumer polling system
  16. LE MOIS DU CLOUD APP 16 Kafka clients and the

    consumer polling system
  17. LE MOIS DU CLOUD APP 17 Kafka clients and the

    consumer polling system
  18. LE MOIS DU CLOUD APP 18 Kafka clients and the

    consumer polling system
  19. LE MOIS DU CLOUD APP 19 Kafka clients and the

    consumer polling system
  20. LE MOIS DU CLOUD Streaming Apps Kafka-Streams and the consumer

    protocol 20
  21. LE MOIS DU CLOUD topic-partition-0 topic-partition-1 topic-partition-2 topic-partition-N APP ◇

    Every topic in Kafka is split into one or more partitions ◇ All the streaming tasks are executed through one or multiple threads of the same instance 21 Kafka-Streams and the consumer protocol
  22. LE MOIS DU CLOUD topic-partition-0 topic-partition-1 topic-partition-2 topic-partition-N APP APP

    ◇ Consumers from the same consumer group cooperate to consume data from topics. ◇ Every instance by joining the group triggers a partition rebalance. 22 Kafka-Streams and the consumer protocol
  23. LE MOIS DU CLOUD Kafka-Streams and the consumer protocol topic-partition-0

    topic-partition-1 topic-partition-2 topic-partition-N APP APP APP APP ◇ The maximum parallelism is determined by the number of partitions of the input topic(s) 23
  24. LE MOIS DU CLOUD Container Orchestration Kubernetes, or the state

    of the art 24
  25. LE MOIS DU CLOUD Container Orchestration: K8s or the state

    of the art ➔ Source: Kubernetes.io Documentation 25
  26. LE MOIS DU CLOUD Container Orchestration Support for custom metrics

    26
  27. LE MOIS DU CLOUD K8s: Support for custom metrics kind:

    Deployment # deployment.yaml #... template: containers: - name: streaming-app # ... - name: prometheus-to-sd # ... adapter.yaml - name: custom-metrics-sd-adapter Your Streaming App Prometheus to Stackdriver https://gcr.io/google-containers/prometheus-to-sd Metrics Server https://gcr.io/google-containers/custom-metrics-stackdriver-adapter JMX metrics in a Prometheus format Stackdriver 27
  28. LE MOIS DU CLOUD Stackdriver, for a full observability of

    your app Monitoring and management for services, containers, applications, and infrastructure • It’s Modular (Monitoring, Logging, Trace … ) • It’s Full-stack insights • It’s Multi-cloud • … and has a rich set of integration tools 28
  29. LE MOIS DU CLOUD K8s: Support for custom metrics kind:

    Deployment # deployment.yaml #... template: containers: - name: streaming-app # ... - name: prometheus-to-sd # ... adapter.yaml - name: custom-metrics-sd-adapter Your Streaming App Prometheus to Stackdriver https://gcr.io/google-containers/prometheus-to-sd Metrics Server https://gcr.io/google-containers/custom-metrics-stackdriver-adapter JMX metrics in a Prometheus format Stackdriver 29
  30. LE MOIS DU CLOUD K8s: Support for custom metrics #

    jmx-exporter.conf --- global: scrape_interval: 1s evaluation_interval: 1s rules: - pattern: "kafka.consumer<type=..., topic=GAME-FRAME-RS, partition=(.*)><>(.*):(.*)" labels: { partition: $2, topic: GAME-FRAME-RS, metric: $3 } name: "consumer_lag_rs" type: GAUGE - pattern: "kafka.consumer<type=..., topic=GAME-FRAME-RQ, partition=(.*)><>(.*):(.*)" labels: { partition: $2, topic: GAME-FRAME-RQ, metric: $3 } name: "consumer_lag_rq" type: GAUGE 30
  31. LE MOIS DU CLOUD K8s: Support for custom metrics $

    curl -s localhost:9001 |grep -E --color=always \ "metric|topic|partition" consumer_lag_rq{client="...",metric="records-lag",partition="0",topic="GAME-FRAME-RQ",} 314562.0 consumer_lag_rq{client="...",metric="records-lag",partition="1",topic="GAME-FRAME-RQ",} 291183.0 consumer_lag_rq{client="...",metric="records-lag",partition="2",topic="GAME-FRAME-RQ",} 302551.0 consumer_lag_rq{client="...",metric="records-lag",partition="3",topic="GAME-FRAME-RQ",} 7319.0 consumer_lag_rq{client="...",metric="records-lag",partition="4",topic="GAME-FRAME-RQ",} 19329.0 consumer_lag_rq{client="...",metric="records-lag",partition="5",topic="GAME-FRAME-RQ",} 30698.0 consumer_lag_rs{client="...",metric="records-lag",partition="0",topic="GAME-FRAME-RS",} 312124.0 consumer_lag_rs{client="...",metric="records-lag",partition="1",topic="GAME-FRAME-RS",} 293837.0 consumer_lag_rs{client="...",metric="records-lag",partition="2",topic="GAME-FRAME-RS",} 302940.0 consumer_lag_rs{client="...",metric="records-lag",partition="3",topic="GAME-FRAME-RS",} 6594.0 consumer_lag_rs{client="...",metric="records-lag",partition="4",topic="GAME-FRAME-RS",} 15777.0 consumer_lag_rs{client="...",metric="records-lag",partition="5",topic="GAME-FRAME-RS",} 24881.0 31
  32. LE MOIS DU CLOUD K8s: Support for custom metrics 1

    With a build tool, create a Docker image of your streaming application lolo ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date 32
  33. LE MOIS DU CLOUD K8s: Support for custom metrics lolo

    ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary 2 Ensure you have a kubernetes cluster ready! Use a infra as code software… like terraform 33
  34. LE MOIS DU CLOUD K8s: Support for custom metrics lolo

    ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml 3 With kubectl create your configurations, secrets, pods and services 34
  35. LE MOIS DU CLOUD K8s: Support for custom metrics lolo

    ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 4 Enjoy your deployment! 35
  36. LE MOIS DU CLOUD K8s: Support for custom metrics lolo

    ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 36
  37. LE MOIS DU CLOUD K8s: Support for custom metrics lolo

    ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 37
  38. LE MOIS DU CLOUD K8s: Support for custom metrics lolo

    ➜ ./gradlew dockerPush <=========----> 73% EXECUTING [2s] > :docker … BUILD SUCCESSFUL in 14s 10 actionable tasks: 5 executed, 5 up-to-date lolo ➜ terraform apply … + google_container_cluster.primary lolo ➜ kubectl create -f deployment.yaml lolo ➜ kubectl get pods prometheus-to-sd kstreams-app 38
  39. LE MOIS DU CLOUD K8s: Support for custom metrics kind:

    Deployment # deployment.yaml #... template: containers: - name: streaming-app # ... - name: prometheus-to-sd # ... adapter.yaml - name: custom-metrics-sd-adapter Your Streaming App Prometheus to Stackdriver https://gcr.io/google-containers/prometheus-to-sd Metrics Server https://gcr.io/google-containers/custom-metrics-stackdriver-adapter Stackdriver 39
  40. LE MOIS DU CLOUD Container Orchestration Horizontal Pod Autoscaler 40

  41. LE MOIS DU CLOUD K8s: Horizontal Pod Autoscaler 41 ➔

    Source: Kubernetes.io Documentation
  42. LE MOIS DU CLOUD K8s: Horizontal Pod Autoscaler 42 -

    Kubernetes Resource - Periodically adjusts the number of replicas - Base on CPU usage in autoscaling/v1 - Memory and custom metrics are covered by the autoscaling/v2beta1 - Use the metrics.k8s.io API through a metric server ➔ Source: Kubernetes.io Documentation
  43. LE MOIS DU CLOUD Scale In / scale out with

    kafka-streams and k8s 43
  44. LE MOIS DU CLOUD 44

  45. LE MOIS DU CLOUD Scale In / scale out with

    kafka-streams and k8s 45
  46. 46 CONCLUSION States migration, changelog compaction, topology upgrades and k8s

    StateFull Sets adoption are the next challenges to ease auto-scaling BUILD THE FUTURE 1. Kafka-Streams exposes relevant metrics related to stream processing 2. Consumer-lag is one of the key metrics to monitor in real time application 3. The cloud trends brings a set of powerful tools on which the Kafka community keep a close look
  47. LE MOIS DU CLOUD 47 MERCI

  48. LE MOIS DU CLOUD 48

  49. LE MOIS DU CLOUD 49 ANNEXES

  50. LE MOIS DU CLOUD 50

  51. LE MOIS DU CLOUD 51 The Horizontal Pod Autoscaler algorithm

    depends on the current metric value and replica number desiredReplicas = ceil[currentReplicas * ( currentMetricValue / desiredMetricValue )] ➢ A ratio of two will double the number of intances within the respect of maxReplicas ➢ By using targetAverageValue, the metric is computed by taking the average of the given metric across all Pods The number of replicas may fluctuating frequently due to the dynamic nature of the metrics, it’s called trashing ➢ --horizontal-pod-autoscaler-downscale-delay (default 5m0s) ➢ --horizontal-pod-autoscaler-upscale-delay (default 3m0s) Note: Both Kafka-Streams topology modification and HPA makes rolling update imposible
  52. LE MOIS DU CLOUD 52 now supports more than https://www.confluent.io/blog/apache-kafka-supports-200k-partitions-per-cluster

    200K partitions
  53. LE MOIS DU CLOUD 53

  54. 54 Links and references https://www.confluent.io/kafka-summit-sf18/deploying- kafka-streams-applications https://www.youtube.com/watch?v=9cyXXmRlGWQ https://www.confluent.io/blog/apache-kafka-kubernetes -could-you-should-you https://kubernetes.io/blog/2018/04/13/local-persistent-v

    olumes-beta/ https://stackoverflow.com/questions/49482873/how-to -deploy-kafka-stream-applications-on-kubernetes https://www.youtube.com/watch?v=9TOoThIKafo&list=P LhMG-8t0efEvJM5Bt2_zNLNVCEWcfRN0i https://www.confluent.io/blog/streaming-in-the-clouds- where-to-start Ideas from The event oriented architecture Concepts from Streams and tables white paper All the code on github DivLoic/xke-kingof-scaling Pictures: ◇ Stormtroopers Photo by Corey Motta on Unsplash ◇ Sreens from https://kubernetes.io/docs/tasks/ ◇ Coin-op machine by xoxo from the Noun Project ◇ Arcade game by Icons Producer from the Noun Project ◇ Mainframe by monkik from the Noun Project