Infrastructure testing using Kubernetes

2cdc5c3059c40c76d5ca7bec8d149f9e?s=47 Ran Tavory
February 24, 2019

Infrastructure testing using Kubernetes

Testing Kafka replication using uReplicator, using Kubernetes, Prometheus, Grafana and friends

2cdc5c3059c40c76d5ca7bec8d149f9e?s=128

Ran Tavory

February 24, 2019
Tweet

Transcript

  1. Infrastructure testing using Kubernetes @rantav

  2. Infrastructure testing using Kubernetes OR: Testing Kafka replication over the

    Atlantic using Golang, Kubernetes and friends
  3. THE GOAL Test Kafka replication across the Atlantic

  4. Hello! I am Ran Tavory I work on multi-region You

    can find me @rantav (slack/twitter/email) 4
  5. The Goal: Test Kafka Replication ⊙ Replicate 30MB - 300MB

    of traffic per second ⊙ Validate correctness ⊙ Measure latency ⊙ Run test cases ◦ Broker crash, cluster resize, packet loss... 5
  6. HOW ? High level design 6

  7. Kafka Replication From https://github.com/AppsFlyer/kafka-mirror-tester 7

  8. With a home-grown producer and consumer 8

  9. WHAT DO WE WANT? 9 DOCUMENTATION

  10. WHAT DO WE WANT? 10 REPRODUCIBILITY

  11. WHAT DO WE WANT? 11 MONITORING

  12. KUBERNETES 12

  13. You keep saying this word Kubernetes What does it mean?

    13
  14. A brief introduction to Kubernetes 14

  15. CLUSTER MANAGER Kubernetes is a 15

  16. NODES The host running a kubelet and a set of

    Pods 16 It manages
  17. PODS Pods consist of 1 or more containers 17 Nodes

    contain
  18. DEPLOYMENTS A set of stateless pods 18 Pods run

  19. STATEFUL SET A set of stateful pods 19 Pods also

    run
  20. KUBECTL The k8s CLI 20 You control k8s using

  21. $ make k8s-all 22

  22. What does that do? ⊙ Provision VMs in AWS ⊙

    Setup VPCs, Subnets, Routing Tables ⊙ Create Security Groups ⊙ Setup Load Balancers ⊙ Install Kubernetes (etcd, masters, nodes) ⊙ Setup Monitoring (Prometheus & Grafana and install dashboard) ⊙ Install Weave Scope ⊙ Install Kafkas (and test them) ⊙ Install uReplicator (and test it) ⊙ Install test programs ⊙ And more… (ASGs, DHCP etc) 23
  23. $ kops create cluster \ --zones us-east-1a,us-east-1b,us-east-1c \ --node-count 40

    \ --node-size i3.large \ --master-size m4.large \ --master-zones us-east-1a \ --networking calico \ --cloud aws 24
  24. What are we building? us-east-1 25 eu-west-1

  25. What are we building? us-east-1 40 nodes k8s cluster 26

    eu-west-1 48 nodes k8s cluster
  26. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 27 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster
  27. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 28 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator
  28. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 10 producers 29 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator 4 consumers
  29. 30 End Result

  30. 31 Grafana Dashboard

  31. 32 Prometheus

  32. 33 Weave Scope

  33. 34 Weave Scope

  34. 35 Weave Scope

  35. 36 kubectl example

  36. TEST IT. Let’s put all this to test now 37

  37. $ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ delete

    pod kafka-source-2 38 KILL A BROKER
  38. 39 KILL A BROKER

  39. $ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ scale

    \ statefulset kafka-source \ --replicas 29 40 RESIZE KAFKA CLUSTER
  40. 41 RESIZE KAFKA CLUSTER

  41. $ kubectl \ --context eu-west-1.k8s.local \ -n ureplicator \ scale

    deployment \ ureplicator-worker \ --replicas 9 42 ADD uREPLICATOR WORKER
  42. 43 ADD uREPLICATOR WORKER

  43. $ make k8s-redeploy-tests 44 ADD NEW TOPIC

  44. 45 ADD NEW TOPIC

  45. $ make k8s-kafka-shell-source $ bin/kafka-topics.sh --zookeeper zookeeper:2181 --alter --topic topic5

    --partitions 300 46 ADD PARTITIONS
  46. $ kubectl --context eu-west-1.k8s.local \ -n ureplicator port-forward \ ureplicator-controller-76ff85b889-l9mzl

    9000 $ curl -X DELETE http://localhost:9000/topics/topic5 $ curl -X POST -d \ '{"topic":"topic5", "numPartitions":"300"}' \ http://localhost:9000/topics 47 ADD PARTITIONS (cont)
  47. 48 PACKET LOSS

  48. 49 PACKET LOSS

  49. CODE DEEP DIVE Kubernetes, head first (just a few bites)

    50
  50. 51 Kafka kind: StatefulSet replicas: 30

  51. Kafka Main container 52 image ports resources probes

  52. 53 Kafka config

  53. 54 Metrics sidecar container

  54. 55 Kafka anti- affinity

  55. 56 uReplicator deployment

  56. 57 Producer deployment

  57. 58 Consumer deployment

  58. 59 Service monitoring

  59. CODE DEEP DIVE Golang Producer/Consumer (just a few bites) 60

  60. 61 Producer Main loop

  61. 62 Producer channels p.ProduceChannel() <- m

  62. 63 Consumer Main loop Loop until: signaled || done. Process

    event Type switch
  63. 64 Consumer Process message

  64. “ Question: How can multiple consumers validate message arrival order?

    65
  65. 66 Producer Sequence numbers messageKey = seq % partitions perKeySeq

    = seq / partitions
  66. $ make k8s-delete-all 67

  67. 68

  68. Thanks! Any questions? You can find me at @rantav &

    rantav@appsflyer.com 69 This presentation: https://speakerdeck.com/rantav/infrastructure-testing-using-kubernetes