An introduce to K8S and Kafka
HOW WE BUILDMICROSERVICESWITH KUBERNETES & KAFKASDN X CLOUD NATIVE MEETUP #20https://speakerdeck.com/chenyunchen/redis-lua-script-with-red-envelope-and-message-queue
View Slide
WHO AM IChen Yun Chen (Alex)> [email protected]> blog.yunchen.twExperience> Backend Engineer atJKOPay> Software Engineer atLinker Networks
KUBERNETESቘ薹硌獅膏褖ګ
CONTAINER ቘ薹膏褖ګ> Container != Docker> Appᥝெ讕瞥獤ग़㮆Dockerfile?> Log෭扮ইຎ茴褸?> 緳矒᯿珸ᥝெ讕ڣ䥁?> 瑊絑ह牏܋羷ፘ狅䷱㺔氂?4
KUBERNETES 硌獅膏褖ګ> Storage: CSI (Container Storage Interface)吖瑿猋ൔ, 梊秚ګ, 盠ᆙ, ᯿蕦虻碘㳫ᴻ?> Network: CNI (Container Network Interface)ๅṛጱ窕ᰁ, ๅ犵ጱ皤螛, ๅग़ጱ翕᪠Օᶎ?> Computing: CRI (Container Runtime Interface)犋አDocker? ೭㬵᪒VM? 窾ݳContainer&VM?5
ܻত礍䯤> 㻌ݣVM䁆ᤈ㮆誢> ֵአᒫӣොጱRedis, MySQL๐率6
疩獈K8S㪔瞥獤๐率犥㻌Ӟ℄讨傶ֺ繸ෝ缰ଏ牧蟂℄奞℄ኼAuth Server> ض疥ጭ獈藨挨๐率ು櫝Logic Server> ݳဩጱ藶穩盅蝑狶蒂ቘ7
疩獈WEBSOCKET蝫娄Auth Server> 奈TCP蝫娄 ׀Android, iOS藨挨WebSocket Server> WebSocket蝫娄 ׀Web藨挨翕殷粚氃क़螵蜉襑穩8
瞥獤LOGIC๐率Http Server> ׀獉蟂ٌਙ㿁褧篷ဩ蝚螂gRPCොୗፗ蝫LogicAvatar Server> 蕣狶㮆Ո牏ᗭ奲瑽猟Message Center Server> 懱௳๐率ፘ橕File Server> 䲆礯๐率ፘ橕牏ݱ圵疳ੑ婘瑽蕣狶9
KAFKABUILDING REAL-TIME DATA PIPELINES
KAFKA PRODUCERProducer> client毆戔䟖ṛރݺֵአොୗ(ܨᔴ坌懱௳ٚ蝑ڊ牧IM皤螛斃硵眤襑藲碉)Topic> Partition碍ᰁӤ褖ݢ犥ض懪2000(憙ݱ蜃ᓕ๐率ಅ׀ጱӤ褖)Partition> client毆戔ྯ稞䌃獈ฎ狅ଧPartition 0→1→2(憙襑穩藲碉౮ᓒkeyጱHash)11
KAFKA CONSUMERConsumerྯ稞襑ޞᎣ托奞ጱ襑穩> Topic> Partition ID> OffsetConsumer GroupݱPartition虻碘ݝ蒂ቘӞ稞> Topic12
疩獈KAFKA䌃獈猻ض蝱Kafkasync捝玲ᆙଉ蚎grpcො礯(䧙ਧ)Consumer> 朼䌘ݱUser牏ग़蕕ᗝ虻碘藶穩Consumer Group> 朼䌘Worker狶瞙稞蒂ቘ13
癲獈ٌਙᒫӣො๐率MySQL> 㮆Ո牏ᗭ奲虻碘缛Redis> 盠玲现夵袅瞂懿犥ے蝧疨ತmongoDB> 瞱ԋ玕懱௳14
ٌਙ襑穩HTTP Proxy Server: Squid> K8S℄讨狅襑穩ᛔ㵕ी仂ֵ藶穩IP虋㵕> ℂ襑ᒒࢧ苽瑿ᰂ窕๐率襑穩(ጮݷ㻌秚ګ)> Develop, Staging, RC 絑ह缛㬵რ藶穩介手15
ࢧ毣㬵፡፡K8Sঅ蒂Service> 瞥獤๐率ኧService狶ፘ传蝢犋ࣁNode樌ጱ翕᪠现Podஂ種樌ই֜ਂ玲Ingress> 蝚螂routingጱොୗ㬵ᓕቘ襑ᥝ䌘क़ู襷ጱService16
MIDDLEWARE ቘ薹Proxy typical load-balancing> SSL termination> ኧReverse Proxy Server究ਧፓ秂17
SERVICE ܻቘtransparent tcp traffic proxy> Clientᒒ狒瞱ݶӞ礿TCP蝫娄> 萞ኧፗ矑硬虋۱֖ᗝ螈౮18
SERVICE HEADLESSDNS ClientEasy solve L7 load-balancing> K8S Service Headless玲Podڜ蔭> Clientᒒ究ਧ揗斉ଘᤍ֖ᗝ19
SERVICE MESH禊盢Ӥݢ䋿֢ෝ覍瑊絑हᛔᤈᓕቘ虻碘窕牧犋አK8Sܻতො礯ፓ獮๚䟖አService Meshෝ๐率ӾData Plane> ݢ犥磦矦Proxy䋿֢> 膏茴褸logፘݶ緡犥sidecar秇ୗControl Plane> Proxy矒ګ> 狅窕ᰁ硈ਧᒽኼ> 蛪犩藨戣20
CI / CDਧ嬝窕纷膏䋿֢
JENKINSFILE ਧ嬝PR窕纷> Prepareୌ缏瑿絑ह㪔㬙ᗟᒫӣො๐率(MySQL, Redis, mongoDB)㪔蝢ᎣGitlab樄তPR窕纷牧纔ྊݳ㬫Branch> Testing䁆ᤈUnit Test㪔叨ኞ介手䁭ޞ> Build Imageଘᤈୌ缏ಅ磪盏๐率ጱimage> Deploy Local Service蟂ᗟಅ磪盏๐率ᛗ瑿K8S Cluster> Reportݳ㬫ಅ磪褩ྦྷጱXML介手䁭ޞ㪔ԻᛗJenkins> Notification咳蝑窕纷奾ຎᛗSlack㪔蝢ᎣGitlab奾PR窕纷牧狅䁭ޞฎ玽ݶݳ㬫22
JENKINSFILE ਧ嬝㬙ᗟ> Deploy Cloud Service疥ض獮ಅ磪盏๐率ጱImageӤ㯽ᛗPrivate Registery㪔蟂ᗟಅ磪盏๐率ᛗ䌘䛑Branch絑हጱK8S Cluster> Notification咳蝑窕纷奾ຎᛗSlack㪔狅奾ຎฎ玽ᥝ䔶ګ蝢Ꭳࣁ娄ጱ姘螀Ո㹓ૡ纷䒍(舙०硻K8S蛪䨝狶ࢧ笔)23
NEXT?GitLab + Jira 碉ݳ萞ኧ秂戢Jira Task翥蒈ᛗGitLabJira疰胼ࣁGitLab獤ඪcommit, merge碻ᛔ㵕疥Taskᑏᛗ蒂ቘӾ牏૪ਠ౮ਠ珿ጱ㻌֖介手窩荠ሲ犋㰍㰍ฎᥝ蝢螂Unit Testইຎྯ稞ݳ㬫獮碉誢介手窩荠ሲ๚Ӥ܋Ӟ䰬纔ྊPR狶ݳ㬫Module Proxy Serverྯ稞叨ኞጱ獊碝介手絑ह襑ᥝӞ㮆揗揣CacheᒫӣොॺկጱProxy Serverslack bot犋ٚ㰍㰍ݝฎ蝢Ꭳ 礍戔Bot Serverಋ㵕᯿珸๐率牏娄Ӥ碍硁翄懯奾ຎ萞ኧ翄ӞࣁSlackՕᶎӤ砺֢㬵糷玲ݱୗ䁭ޞ24
MORE CHALLENGECRD(Custom Resource Definitions)> 疥ग़๐率ਧ嬝傶㻌Ӟ虻რ硯獈K8S or VM> 仂戔ਧ䲆ᓕቘ犖仂०藮> 姘螀Ո㹓ๅ胼䌕ဳෝ碉誢๐率ጱᓕቘ25
Q & A