Infrastructure Testing Using Kubernetes And Golang

1. Infrastructure testing using Kubernetes @rantav
2. OR: Testing Kafka replication over the Atlantic using Golang, Kubernetes and friends
3. Hello! I am Ran Tavory I work on multi-region at appsflyer You can find me @rantav (twitter/gmail/facebook) 3
4. The Goal: Test Kafka Replication ⊙ Replicate 30MB - 300MB of traffic per second ⊙ Validate correctness ⊙ Measure latency ⊙ Run test cases (failure scenarios) ◦ Broker crash, cluster resize, packet loss... 4
5. Kafka ? Queueing System Stream Processing Brokers Partitions 5
6. HOW ? High level design 6
7. Kafka Replication From https://github.com/AppsFlyer/kafka-mirror-tester 7
8. With a home-grown producer and consumer written in golang 8
9. WHAT DO WE WANT? 9 DOCUMENTATION HOW DO WE DO IT?
10. WHAT DO WE WANT? 10 REPRODUCIBILITY HOW DO WE DO IT?
11. WHAT DO WE WANT? 11 MONITORING HOW DO WE DO IT?
12. KUBERNETES 12
13. You keep saying this word Kubernetes What does it mean? 13
14. A brief introduction to Kubernetes 14
15. CLUSTER MANAGER Kubernetes is a 15
16. NODES The host running a kubelet and a set of Pods 16 It manages
17. PODS Pods consist of 1 or more containers 17 Nodes contain
18. DEPLOYMENTS A set of stateless pods 18 Pods run
19. STATEFUL SET A set of stateful pods 19 Pods also run
20. KUBECTL The k8s CLI 20 You control k8s using
21. “
22. $ make k8s-all 22
23. What does that do? ⊙ Provision VMs in AWS ⊙ Setup VPCs, Subnets, Routing Tables ⊙ Create Security Groups ⊙ Setup Load Balancers ⊙ Install Kubernetes (etcd, masters, nodes) ⊙ Setup Monitoring (Prometheus & Grafana and install dashboard) ⊙ Install Weave Scope ⊙ Install Kafkas (and test them) ⊙ Install uReplicator (and test it) ⊙ Install test programs ⊙ And more… (ASGs, DHCP etc) 23
24. $ kops create cluster \ --zones us-east-1a,us-east-1b,us-east-1c \ --node-count 40 \ --node-size i3.large \ --master-size m4.large \ --master-zones us-east-1a \ --networking calico \ --cloud aws 24
25. What are we building? us-east-1 25 eu-west-1
26. What are we building? us-east-1 40 nodes k8s cluster 26 eu-west-1 48 nodes k8s cluster
27. What are we building? us-east-1 40 nodes k8s cluster 30 brokers kafka cluster 27 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster
28. What are we building? us-east-1 40 nodes k8s cluster 30 brokers kafka cluster 28 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator
29. What are we building? us-east-1 40 nodes k8s cluster 30 brokers kafka cluster 10 producers 29 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator 4 consumers
30. 30 End Result
31. 31 Grafana Dashboard
32. 32 Prometheus
33. 33 Weave Scope
34. 34 Weave Scope
35. 35 Weave Scope
36. 36 kubectl example
37. TEST IT. Let’s put all this to test now 37
38. $ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ delete pod kafka-source-2 38 KILL A BROKER
39. 39 KILL A BROKER
40. $ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ scale \ statefulset kafka-source \ --replicas 29 40 RESIZE KAFKA CLUSTER
41. 41 RESIZE KAFKA CLUSTER
42. $ kubectl \ --context eu-west-1.k8s.local \ -n ureplicator \ scale deployment \ ureplicator-worker \ --replicas 9 42 ADD uREPLICATOR WORKER
43. 43 ADD uREPLICATOR WORKER
44. $ make k8s-redeploy-tests 44 ADD NEW TOPIC
45. 45 ADD NEW TOPIC
46. $ make k8s-kafka-shell-source $ bin/kafka-topics.sh --zookeeper zookeeper:2181 --alter --topic topic5 --partitions 300 46 ADD PARTITIONS
47. $ kubectl --context eu-west-1.k8s.local \ -n ureplicator port-forward \ ureplicator-controller-76ff85b889-l9mzl 9000 $ curl -X DELETE http://localhost:9000/topics/topic5 $ curl -X POST -d \ '{"topic":"topic5", "numPartitions":"300"}' \ http://localhost:9000/topics 47 ADD PARTITIONS (cont)
48. 48 PACKET LOSS
49. 49 PACKET LOSS
50. CODE DEEP DIVE Golang Producer/Consumer (just a few bites) 50
51. 51 Producer Main loop
52. 52 Rate limit in Golang The classic gobyexample
53. 53 Rate limit in Golang The classic gobyexample What is the problem here?
54. 54 Rate limit in Golang Solution: Use package rate
55. 55 Rate limit in Golang Other solutions: Juju and Uber
56. 56 Producer: using channels and the c library librdkafka p.ProduceChannel() <- m
57. 57 Consumer Main loop Loop select until: signaled || done. Process event
58. 58 Consumer Process message
59. “ Question: How can multiple consumers validate message arrival order? 59
60. 60 Producer Sequence numbers messageKey = seq % partitions perKeySeq = seq / partitions
61. CODE DEEP DIVE Kubernetes, head first (just a few bites) 61
62. 62 Kafka kind: StatefulSet replicas: 30
63. Kafka Main container 63 image ports resources probes
64. 64 Kafka config
65. 65 Metrics sidecar container
66. 66 Kafka anti- affinity
67. 67 uReplicator deployment
68. 68 Producer deployment
69. 69 Consumer deployment
70. 70 Service monitoring
71. $ make k8s-delete-all 71
73. Thanks! Any questions? You can find me at @rantav & rantav@appsflyer.com 73 This presentation: https://speakerdeck.com/rantav/infrastructure-testing-using-kubernetes- and-go The project: https://github.com/AppsFlyer/kafka-mirror-tester