Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infrastructure Testing Using Kubernetes And Gol...

Infrastructure Testing Using Kubernetes And Golang

Avatar for Ran Tavory

Ran Tavory

May 06, 2019
Tweet

More Decks by Ran Tavory

Other Decks in Technology

Transcript

  1. Hello! I am Ran Tavory I work on multi-region at

    appsflyer You can find me @rantav (twitter/gmail/facebook) 3
  2. The Goal: Test Kafka Replication ⊙ Replicate 30MB - 300MB

    of traffic per second ⊙ Validate correctness ⊙ Measure latency ⊙ Run test cases (failure scenarios) ◦ Broker crash, cluster resize, packet loss... 4
  3. What does that do? ⊙ Provision VMs in AWS ⊙

    Setup VPCs, Subnets, Routing Tables ⊙ Create Security Groups ⊙ Setup Load Balancers ⊙ Install Kubernetes (etcd, masters, nodes) ⊙ Setup Monitoring (Prometheus & Grafana and install dashboard) ⊙ Install Weave Scope ⊙ Install Kafkas (and test them) ⊙ Install uReplicator (and test it) ⊙ Install test programs ⊙ And more… (ASGs, DHCP etc) 23
  4. $ kops create cluster \ --zones us-east-1a,us-east-1b,us-east-1c \ --node-count 40

    \ --node-size i3.large \ --master-size m4.large \ --master-zones us-east-1a \ --networking calico \ --cloud aws 24
  5. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 27 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster
  6. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 28 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator
  7. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 10 producers 29 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator 4 consumers
  8. $ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ scale

    \ statefulset kafka-source \ --replicas 29 40 RESIZE KAFKA CLUSTER
  9. $ kubectl \ --context eu-west-1.k8s.local \ -n ureplicator \ scale

    deployment \ ureplicator-worker \ --replicas 9 42 ADD uREPLICATOR WORKER
  10. $ kubectl --context eu-west-1.k8s.local \ -n ureplicator port-forward \ ureplicator-controller-76ff85b889-l9mzl

    9000 $ curl -X DELETE http://localhost:9000/topics/topic5 $ curl -X POST -d \ '{"topic":"topic5", "numPartitions":"300"}' \ http://localhost:9000/topics 47 ADD PARTITIONS (cont)
  11. 72

  12. Thanks! Any questions? You can find me at @rantav &

    rantav@appsflyer.com 73 This presentation: https://speakerdeck.com/rantav/infrastructure-testing-using-kubernetes- and-go The project: https://github.com/AppsFlyer/kafka-mirror-tester