Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Infrastructure testing using Kubernetes
Search
Ran Tavory
February 24, 2019
Programming
0
540
Infrastructure testing using Kubernetes
Testing Kafka replication using uReplicator, using Kubernetes, Prometheus, Grafana and friends
Ran Tavory
February 24, 2019
Tweet
Share
More Decks by Ran Tavory
See All by Ran Tavory
go-grpc-channelz: a Go based UI for gRPC's channelz
rantav
0
580
Code Review (Navy Hackathon)
rantav
0
73
Infrastructure Testing Using Kubernetes And Golang
rantav
0
80
Interview Workshop - Technical Questions
rantav
0
330
Code Review Best Practices
rantav
0
83
Code Review @ AppsFlyer
rantav
0
84
GraphQL at Yodas
rantav
2
170
Git for champs
rantav
0
140
Code Review at Yodas
rantav
0
180
Other Decks in Programming
See All in Programming
AIと”コードの評価関数”を共有する / Share the "code evaluation function" with AI
euglena1215
1
190
ソフトウェア設計とAI技術の活用
masuda220
PRO
25
6.5k
レベル1の開発生産性向上に取り組む − 日々の作業の効率化・自動化を通じた改善活動
kesoji
1
340
PHPUnitの限界をPlaywrightで補完するテストアプローチ
yuzneri
0
260
Vibe Codingの幻想を超えて-生成AIを現場で使えるようにするまでの泥臭い話.ai
fumiyakume
18
8.9k
MDN Web Docs に日本語翻訳でコントリビュートしたくなる
ohmori_yusuke
1
130
Claude Code + Container Use と Cursor で作る ローカル並列開発環境のススメ / ccc local dev
kaelaela
12
7.3k
AWS Summit Japan 2024と2025の比較/はじめてのKiro、今あなたは岐路に立つ
satoshi256kbyte
1
220
CDK引数設計道場100本ノック
badmintoncryer
2
530
Rails Frontend Evolution: It Was a Setup All Along
skryukov
0
310
ソフトウェア品質を数字で捉える技術。事業成長を支えるシステム品質の マネジメント
takuya542
2
16k
[SRE NEXT] 複雑なシステムにおけるUser Journey SLOの導入
yakenji
0
650
Featured
See All Featured
Rebuilding a faster, lazier Slack
samanthasiow
83
9.1k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.7k
Fireside Chat
paigeccino
37
3.5k
Automating Front-end Workflow
addyosmani
1370
200k
Building an army of robots
kneath
306
45k
A Tale of Four Properties
chriscoyier
160
23k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
357
30k
Art, The Web, and Tiny UX
lynnandtonic
301
21k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
50k
Optimising Largest Contentful Paint
csswizardry
37
3.3k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
130
19k
jQuery: Nuts, Bolts and Bling
dougneiner
63
7.8k
Transcript
Infrastructure testing using Kubernetes @rantav
Infrastructure testing using Kubernetes OR: Testing Kafka replication over the
Atlantic using Golang, Kubernetes and friends
THE GOAL Test Kafka replication across the Atlantic
Hello! I am Ran Tavory I work on multi-region You
can find me @rantav (slack/twitter/email) 4
The Goal: Test Kafka Replication ⊙ Replicate 30MB - 300MB
of traffic per second ⊙ Validate correctness ⊙ Measure latency ⊙ Run test cases ◦ Broker crash, cluster resize, packet loss... 5
HOW ? High level design 6
Kafka Replication From https://github.com/AppsFlyer/kafka-mirror-tester 7
With a home-grown producer and consumer 8
WHAT DO WE WANT? 9 DOCUMENTATION
WHAT DO WE WANT? 10 REPRODUCIBILITY
WHAT DO WE WANT? 11 MONITORING
KUBERNETES 12
You keep saying this word Kubernetes What does it mean?
13
A brief introduction to Kubernetes 14
CLUSTER MANAGER Kubernetes is a 15
NODES The host running a kubelet and a set of
Pods 16 It manages
PODS Pods consist of 1 or more containers 17 Nodes
contain
DEPLOYMENTS A set of stateless pods 18 Pods run
STATEFUL SET A set of stateful pods 19 Pods also
run
KUBECTL The k8s CLI 20 You control k8s using
“
$ make k8s-all 22
What does that do? ⊙ Provision VMs in AWS ⊙
Setup VPCs, Subnets, Routing Tables ⊙ Create Security Groups ⊙ Setup Load Balancers ⊙ Install Kubernetes (etcd, masters, nodes) ⊙ Setup Monitoring (Prometheus & Grafana and install dashboard) ⊙ Install Weave Scope ⊙ Install Kafkas (and test them) ⊙ Install uReplicator (and test it) ⊙ Install test programs ⊙ And more… (ASGs, DHCP etc) 23
$ kops create cluster \ --zones us-east-1a,us-east-1b,us-east-1c \ --node-count 40
\ --node-size i3.large \ --master-size m4.large \ --master-zones us-east-1a \ --networking calico \ --cloud aws 24
What are we building? us-east-1 25 eu-west-1
What are we building? us-east-1 40 nodes k8s cluster 26
eu-west-1 48 nodes k8s cluster
What are we building? us-east-1 40 nodes k8s cluster 30
brokers kafka cluster 27 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster
What are we building? us-east-1 40 nodes k8s cluster 30
brokers kafka cluster 28 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator
What are we building? us-east-1 40 nodes k8s cluster 30
brokers kafka cluster 10 producers 29 eu-west-1 48 nodes k8s cluster 30 brokers kafka cluster 8 workers uReplicator 4 consumers
30 End Result
31 Grafana Dashboard
32 Prometheus
33 Weave Scope
34 Weave Scope
35 Weave Scope
36 kubectl example
TEST IT. Let’s put all this to test now 37
$ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ delete
pod kafka-source-2 38 KILL A BROKER
39 KILL A BROKER
$ kubectl \ --context us-east-1.k8s.local \ -n kafka-source \ scale
\ statefulset kafka-source \ --replicas 29 40 RESIZE KAFKA CLUSTER
41 RESIZE KAFKA CLUSTER
$ kubectl \ --context eu-west-1.k8s.local \ -n ureplicator \ scale
deployment \ ureplicator-worker \ --replicas 9 42 ADD uREPLICATOR WORKER
43 ADD uREPLICATOR WORKER
$ make k8s-redeploy-tests 44 ADD NEW TOPIC
45 ADD NEW TOPIC
$ make k8s-kafka-shell-source $ bin/kafka-topics.sh --zookeeper zookeeper:2181 --alter --topic topic5
--partitions 300 46 ADD PARTITIONS
$ kubectl --context eu-west-1.k8s.local \ -n ureplicator port-forward \ ureplicator-controller-76ff85b889-l9mzl
9000 $ curl -X DELETE http://localhost:9000/topics/topic5 $ curl -X POST -d \ '{"topic":"topic5", "numPartitions":"300"}' \ http://localhost:9000/topics 47 ADD PARTITIONS (cont)
48 PACKET LOSS
49 PACKET LOSS
CODE DEEP DIVE Kubernetes, head first (just a few bites)
50
51 Kafka kind: StatefulSet replicas: 30
Kafka Main container 52 image ports resources probes
53 Kafka config
54 Metrics sidecar container
55 Kafka anti- affinity
56 uReplicator deployment
57 Producer deployment
58 Consumer deployment
59 Service monitoring
CODE DEEP DIVE Golang Producer/Consumer (just a few bites) 60
61 Producer Main loop
62 Producer channels p.ProduceChannel() <- m
63 Consumer Main loop Loop until: signaled || done. Process
event Type switch
64 Consumer Process message
“ Question: How can multiple consumers validate message arrival order?
65
66 Producer Sequence numbers messageKey = seq % partitions perKeySeq
= seq / partitions
$ make k8s-delete-all 67
68
Thanks! Any questions? You can find me at @rantav &
[email protected]
69 This presentation: https://speakerdeck.com/rantav/infrastructure-testing-using-kubernetes