Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
How we build microservices with Kubernetes & Kafka
Search
Yun Chen
September 26, 2019
Programming
3
380
How we build microservices with Kubernetes & Kafka
An introduce to K8S and Kafka
Yun Chen
September 26, 2019
Tweet
Share
More Decks by Yun Chen
See All by Yun Chen
Redis Lua Script With Red Envelope & Message Queue
chenyunchen
0
600
Kubernetes Library with client-go
chenyunchen
0
420
[Graduate Project] Drupal Upgrade & Docker Deploy
chenyunchen
0
140
WECO Porject
chenyunchen
0
85
From Python Import Fun
chenyunchen
0
400
Docker
chenyunchen
2
200
NISRA fun with burp suite
chenyunchen
1
200
Vim
chenyunchen
0
160
Fun with programming
chenyunchen
0
160
Other Decks in Programming
See All in Programming
React への依存を最小にするフロントエンド設計
takonda
21
8.5k
romajip: 日本の住所CSVデータを活用した英語住所変換ライブラリを作った話
sangunkang
0
2.3k
気をつけたい!Desktop対応で陥りやすい罠とその対策
goto_tsl
0
170
Serverless苦闘史
mosh_inc
0
140
Creating a Free Video Ad Network on the Edge
mizoguchicoji
0
150
Develop iOS apps with Neovim / vimconf_2024
uhooi
1
120
Thoughts and experiences on Rust and TypeScript
unvalley
2
200
CSC509 Lecture 12
javiergs
PRO
0
180
AWS Lambdaから始まった Serverlessの「熱」とキャリアパス / It started with AWS Lambda Serverless “fever” and career path
seike460
PRO
1
410
Leverage LLMs in Java with LangChain4j and Quarkus
hollycummins
0
110
[FlutterKaigi2024] Effective Form 〜Flutterによる複雑なフォーム開発の実践〜
chocoyama
0
3.9k
Enabling DevOps and Team Topologies Through Architecture: Architecting for Fast Flow
cer
PRO
0
420
Featured
See All Featured
Designing on Purpose - Digital PM Summit 2013
jponch
115
7k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
27
4.3k
How to train your dragon (web standard)
notwaldorf
88
5.7k
How STYLIGHT went responsive
nonsquared
95
5.2k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
59k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
10
760
Docker and Python
trallard
40
3.1k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
250
21k
VelocityConf: Rendering Performance Case Studies
addyosmani
326
24k
Measuring & Analyzing Core Web Vitals
bluesmoon
4
150
Testing 201, or: Great Expectations
jmmastey
40
7.1k
Designing for humans not robots
tammielis
250
25k
Transcript
HOW WE BUILD MICROSERVICES WITH KUBERNETES & KAFKA SDN X
CLOUD NATIVE MEETUP #20 https://speakerdeck.com/chenyunchen/redis-lua-script-with-red-envelope-and-message-queue
WHO AM I Chen Yun Chen (Alex) >
[email protected]
>
blog.yunchen.tw Experience > Backend Engineer at JKOPay > Software Engineer at Linker Networks
KUBERNETES ቘ薹硌獅膏褖ګ
CONTAINER ቘ薹膏褖ګ > Container != Docker > Appᥝெ讕瞥獤ग़㮆Dockerfile? > Log෭扮ইຎ茴褸?
> 緳矒᯿珸ᥝெ讕ڣ䥁? > 瑊絑ह牏܋羷ፘ狅䷱㺔氂? 4
KUBERNETES 硌獅膏褖ګ > Storage: CSI (Container Storage Interface) 吖瑿猋ൔ, 梊秚ګ,
盠ᆙ, ᯿蕦虻碘㳫ᴻ? > Network: CNI (Container Network Interface) ๅṛጱ窕ᰁ, ๅ犵ጱ皤螛, ๅग़ጱ翕᪠Օᶎ? > Computing: CRI (Container Runtime Interface) 犋አDocker? ೭㬵᪒VM? 窾ݳContainer&VM? 5
ܻত礍䯤 > 㻌ݣVM䁆ᤈ㮆誢 > ֵአᒫӣොጱRedis, MySQL๐率 6
疩獈K8S㪔瞥獤๐率 犥㻌Ӟ℄讨傶ֺ 繸ෝ缰ଏ牧蟂℄奞℄ኼ Auth Server > ض疥ጭ獈藨挨๐率ು櫝 Logic Server >
ݳဩጱ藶穩盅蝑狶蒂ቘ 7
疩獈WEBSOCKET蝫娄 Auth Server > 奈TCP蝫娄 ׀Android, iOS藨挨 WebSocket Server >
WebSocket蝫娄 ׀Web藨挨 翕殷粚氃क़螵蜉襑穩 8
瞥獤LOGIC๐率 Http Server > ׀獉蟂ٌਙ㿁褧 篷ဩ蝚螂gRPCොୗፗ蝫Logic Avatar Server > 蕣狶㮆Ո牏ᗭ奲瑽猟
Message Center Server > 懱௳๐率ፘ橕 File Server > 䲆礯๐率ፘ橕牏ݱ圵疳ੑ婘瑽蕣狶 9
KAFKA BUILDING REAL-TIME DATA PIPELINES
KAFKA PRODUCER Producer > client毆戔䟖ṛރݺֵአොୗ (ܨᔴ坌懱௳ٚ蝑ڊ牧IM皤螛斃硵眤襑藲碉) Topic > Partition碍ᰁӤ褖ݢ犥ض懪2000 (憙ݱ蜃ᓕ๐率ಅ׀ጱӤ褖)
Partition > client毆戔ྯ稞䌃獈ฎ狅ଧ Partition 0→1→2 (憙襑穩藲碉౮ᓒkeyጱHash) 11
KAFKA CONSUMER Consumer ྯ稞襑ޞᎣ托奞ጱ襑穩 > Topic > Partition ID >
Offset Consumer Group ݱPartition虻碘ݝ蒂ቘӞ稞 > Topic 12
疩獈KAFKA 䌃獈猻ض蝱Kafka sync捝玲ᆙଉ蚎grpcො礯(䧙ਧ) Consumer > 朼䌘ݱUser牏ग़蕕ᗝ虻碘藶穩 Consumer Group > 朼䌘Worker狶瞙稞蒂ቘ
13
癲獈ٌਙᒫӣො๐率 MySQL > 㮆Ո牏ᗭ奲虻碘缛 Redis > 盠玲现夵袅瞂懿犥ے蝧疨ತ mongoDB > 瞱ԋ玕懱௳
14
ٌਙ襑穩 HTTP Proxy Server: Squid > K8S℄讨狅襑穩ᛔ㵕ी仂ֵ藶穩IP虋㵕 > ℂ襑ᒒࢧ苽瑿ᰂ窕๐率襑穩(ጮݷ㻌秚ګ) >
Develop, Staging, RC 絑ह缛㬵რ藶穩介手 15
ࢧ毣㬵፡፡K8Sঅ蒂 Service > 瞥獤๐率ኧService狶ፘ传蝢 犋ࣁNode樌ጱ翕᪠现 Podஂ種樌ই֜ਂ玲 Ingress > 蝚螂routingጱොୗ㬵ᓕቘ 襑ᥝ䌘क़ู襷ጱService
16
MIDDLEWARE ቘ薹 Proxy typical load-balancing > SSL termination > ኧReverse
Proxy Server究ਧፓ秂 17
SERVICE ܻቘ transparent tcp traffic proxy > Clientᒒ狒瞱ݶӞ礿TCP蝫娄 > 萞ኧፗ矑硬虋۱֖ᗝ螈౮
18
SERVICE HEADLESS DNS Client Easy solve L7 load-balancing > K8S
Service Headless玲Podڜ蔭 > Clientᒒ究ਧ揗斉ଘᤍ֖ᗝ 19
SERVICE MESH 禊盢Ӥݢ䋿֢ෝ覍瑊絑ह ᛔᤈᓕቘ虻碘窕牧犋አK8Sܻতො礯 ፓ獮๚䟖አService Meshෝ๐率Ӿ Data Plane > ݢ犥磦矦Proxy䋿֢
> 膏茴褸logፘݶ緡犥sidecar秇ୗ Control Plane > Proxy矒ګ > 狅窕ᰁ硈ਧᒽኼ > 蛪犩藨戣 20
CI / CD ਧ嬝窕纷膏䋿֢
JENKINSFILE ਧ嬝PR窕纷 > Prepare ୌ缏瑿絑ह㪔㬙ᗟᒫӣො๐率 (MySQL, Redis, mongoDB) 㪔蝢ᎣGitlab樄তPR窕纷牧纔ྊݳ㬫Branch >
Testing 䁆ᤈUnit Test㪔叨ኞ介手䁭ޞ > Build Image ଘᤈୌ缏ಅ磪盏๐率ጱimage > Deploy Local Service 蟂ᗟಅ磪盏๐率ᛗ瑿K8S Cluster > Report ݳ㬫ಅ磪褩ྦྷጱXML介手䁭ޞ㪔ԻᛗJenkins > Notification 咳蝑窕纷奾ຎᛗSlack 㪔蝢ᎣGitlab奾PR窕纷牧狅䁭ޞฎ玽ݶݳ㬫 22
JENKINSFILE ਧ嬝㬙ᗟ > Deploy Cloud Service 疥ض獮ಅ磪盏๐率ጱImage Ӥ㯽ᛗPrivate Registery 㪔蟂ᗟಅ磪盏๐率ᛗ
䌘䛑Branch絑हጱK8S Cluster > Notification 咳蝑窕纷奾ຎᛗSlack 㪔狅奾ຎฎ玽ᥝ䔶ګ蝢Ꭳ ࣁ娄ጱ姘螀Ո㹓ૡ纷䒍 (舙०硻K8S蛪䨝狶ࢧ笔) 23
NEXT? GitLab + Jira 碉ݳ 萞ኧ秂戢Jira Task翥蒈ᛗGitLab Jira疰胼ࣁGitLab獤ඪcommit, merge碻 ᛔ㵕疥Taskᑏᛗ蒂ቘӾ牏૪ਠ౮
ਠ珿ጱ㻌֖介手窩荠ሲ 犋㰍㰍ฎᥝ蝢螂Unit Test ইຎྯ稞ݳ㬫獮碉誢介手窩荠ሲ๚Ӥ܋ Ӟ䰬纔ྊPR狶ݳ㬫 Module Proxy Server ྯ稞叨ኞጱ獊碝介手絑ह 襑ᥝӞ㮆揗揣CacheᒫӣොॺկጱProxy Server slack bot 犋ٚ㰍㰍ݝฎ蝢Ꭳ 礍戔Bot Server ಋ㵕᯿珸๐率牏娄Ӥ碍硁翄懯奾ຎ 萞ኧ翄ӞࣁSlackՕᶎӤ砺֢㬵糷玲ݱୗ䁭ޞ 24
MORE CHALLENGE CRD(Custom Resource Definitions) > 疥ग़๐率ਧ嬝傶㻌Ӟ虻რ硯獈K8S or VM >
仂戔ਧ䲆ᓕቘ犖仂०藮 > 姘螀Ո㹓ๅ胼䌕ဳෝ碉誢๐率ጱᓕቘ 25
Q & A