Infrastructure testing using Kubernetes

Transcript

1. Infrastructure testing using Kubernetes @rantav

2. Infrastructure testing using Kubernetes OR: Testing Kafka replication over the

   Atlantic using Golang, Kubernetes and friends

3. THE GOAL Test Kafka replication across the Atlantic

4. Hello! I am Ran Tavory I work on multi-region You

   can find me @rantav (slack/twitter/email) 4

5. The Goal: Test Kafka Replication ⊙ Replicate 30MB - 300MB

   of traffic per second ⊙ Validate correctness ⊙ Measure latency ⊙ Run test cases ◦ Broker crash, cluster resize, packet loss... 5

6. HOW ? High level design 6

7. Kafka Replication From https://github.com/AppsFlyer/kafka-mirror-tester 7

8. With a home-grown producer and consumer 8

9. WHAT DO WE WANT? 9 DOCUMENTATION

10. WHAT DO WE WANT? 10 REPRODUCIBILITY

11. WHAT DO WE WANT? 11 MONITORING

12. KUBERNETES 12

13. You keep saying this word Kubernetes What does it mean?

    13

14. A brief introduction to Kubernetes 14

15. CLUSTER MANAGER Kubernetes is a 15

16. NODES The host running a kubelet and a set of

    Pods 16 It manages

17. PODS Pods consist of 1 or more containers 17 Nodes

    contain

18. DEPLOYMENTS A set of stateless pods 18 Pods run

19. STATEFUL SET A set of stateful pods 19 Pods also

    run

20. KUBECTL The k8s CLI 20 You control k8s using

21. “

22. $ make k8s-all 22

23. What does that do? ⊙ Provision VMs in AWS ⊙

    Setup VPCs, Subnets, Routing Tables ⊙ Create Security Groups ⊙ Setup Load Balancers ⊙ Install Kubernetes (etcd, masters, nodes) ⊙ Setup Monitoring (Prometheus & Grafana and install dashboard) ⊙ Install Weave Scope ⊙ Install Kafkas (and test them) ⊙ Install uReplicator (and test it) ⊙ Install test programs ⊙ And more… (ASGs, DHCP etc) 23

24. $ kops create cluster \ --zones us-east-1a,us-east-1b,us-east-1c \ --node-count 40

    \ --node-size i3.large \ --master-size m4.large \ --master-zones us-east-1a \ --networking calico \ --cloud aws 24

25. What are we building? us-east-1 25 eu-west-1

26. What are we building? us-east-1 40 nodes k8s cluster 26

    eu-west-1 48 nodes k8s cluster

27. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 27 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster

28. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 28 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator

29. What are we building? us-east-1 40 nodes k8s cluster 30

    brokers kafka cluster 10 producers 29 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator 4 consumers

30. 30 End Result

31. 31 Grafana Dashboard

32. 32 Prometheus

33. 33 Weave Scope

34. 34 Weave Scope

35. 35 Weave Scope

36. 36 kubectl example

37. TEST IT. Let’s put all this to test now 37

38. $ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ delete

    pod kafka-source-2 38 KILL A BROKER

39. 39 KILL A BROKER

40. $ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ scale

    \ statefulset kafka-source \ --replicas 29 40 RESIZE KAFKA CLUSTER

41. 41 RESIZE KAFKA CLUSTER

42. $ kubectl \ --context eu-west-1.k8s.local \ -n ureplicator \ scale

    deployment \ ureplicator-worker \ --replicas 9 42 ADD uREPLICATOR WORKER

43. 43 ADD uREPLICATOR WORKER

44. $ make k8s-redeploy-tests 44 ADD NEW TOPIC

45. 45 ADD NEW TOPIC

46. $ make k8s-kafka-shell-source $ bin/kafka-topics.sh --zookeeper zookeeper:2181 --alter --topic topic5

    --partitions 300 46 ADD PARTITIONS

47. $ kubectl --context eu-west-1.k8s.local \ -n ureplicator port-forward \ ureplicator-controller-76ff85b889-l9mzl

    9000 $ curl -X DELETE http://localhost:9000/topics/topic5 $ curl -X POST -d \ '{"topic":"topic5", "numPartitions":"300"}' \ http://localhost:9000/topics 47 ADD PARTITIONS (cont)

48. 48 PACKET LOSS

49. 49 PACKET LOSS

50. CODE DEEP DIVE Kubernetes, head first (just a few bites)

    50

51. 51 Kafka kind: StatefulSet replicas: 30

52. Kafka Main container 52 image ports resources probes

53. 53 Kafka config

54. 54 Metrics sidecar container

55. 55 Kafka anti- affinity

56. 56 uReplicator deployment

57. 57 Producer deployment

58. 58 Consumer deployment

59. 59 Service monitoring

60. CODE DEEP DIVE Golang Producer/Consumer (just a few bites) 60

61. 61 Producer Main loop

62. 62 Producer channels p.ProduceChannel() <- m

63. 63 Consumer Main loop Loop until: signaled || done. Process

    event Type switch

64. 64 Consumer Process message

65. “ Question: How can multiple consumers validate message arrival order?

    65

66. 66 Producer Sequence numbers messageKey = seq % partitions perKeySeq

    = seq / partitions

67. $ make k8s-delete-all 67

68. 68

69. Thanks! Any questions? You can find me at @rantav &

    rantav@appsflyer.com 69 This presentation: https://speakerdeck.com/rantav/infrastructure-testing-using-kubernetes