LINEのPrivate Cloudを支えるデプロイ基盤の課題解決に向けた取り組み / Design of deployment platform for resolving operational issues of LINE's Private Cloud

Slide 1

Slide 1 text

LINEのPrivate Cloudを⽀えるデプロイ基盤の課題解決に向けた取り組み Wataru Manji – LINE Corp. 2021.09

Slide 2

Slide 2 text

LINEのPrivate Cloudに対するデプロイ基盤による課題解決 Wataru Manji – LINE Corp. 2021.09

Slide 3

Slide 3 text

஫ɿ·ͩ΍ͬͯͳ͍

Slide 4

Slide 4 text

ొஃऀ঺հ /55ιϑτ΢ΣΞ ݱ /55ςΫϊΫϩε ೖࣾ ɾ0QFO4UBDLͷݕূɺࣾ಺Ϋϥ΢υ։ൃ ɾ/8ػثͷΦϖϨʔγϣϯޮ཰Խͷݚڀ ɾ/55άϧʔϓͷٕज़ΠϕϯτӡӦ -*/&ೖࣾ 7FSEB3FMJBCJMJUZ&OHJOFFSJOH ɾ؂ࢹํࣜͷݟ௚͠ɺΞϥʔτ੔ཧɺΦϯίʔϧͷಋೖ ɾσϓϩΠํࣜͷվળ ɾͦͷଞϓϩδΣΫτاըͳͲ dϚωʔδϟʔ झຯ͸ےτϨͱࢄา ᤈ࣏ ব .BOBHFS 7FSEB 3FMJBCJMJUZ &OHJOFFSJOH 5FBN !@NBOKJ

Slide 5

Slide 5 text

7FSEBͷ঺հ αʔϏεͷෳࡶੑʹ͍ͭͯ ݱঢ়ͷ՝୊ σϓϩΠʹ૊ࠐΈ͍ͨػೳͱͦΕʹΑͬͯղܾͰ͖Δ͜ͱ LTͷ࢓૊ΈΛ࢖࣮ͬͨ૷ͷΞΠσΞ 7FSEBͰͷ ཁૉٕज़ͷ ಋೖࣄྫ まとめ 01 02 03 04 05 06 07 Contents

Slide 6

Slide 6 text

7FSEBͷ঺հ

Slide 7

Slide 7 text

8IBUJT7FSEB

Slide 8

Slide 8 text

5IFTFSWJDFTPG7FSEB )FSPLV Elasticsearch

Slide 9

Slide 9 text

αʔϏεͷෳࡶੑʹ͍ͭͯ

Slide 10

Slide 10 text

4FSWJDFEFQFOEFODJFT 4PNF7FSEBTFSWJDFTBSFEFQFOEFOUPOPUIFS7FSEBTFSWJDFT

Slide 11

Slide 11 text

%JGGJDVMUJFTJOUSPVCMFTIPPUJOH ɾ3PVUF"EWFSUJTF ɾ/FUXPSLJOHJO)7 ɾ%)$1GPS7. ɾ4FOESFDFJWF31$T l*DBOOPUDSFBUFOFX7.z ʜCVU5IFSFBSFNVMUJQMFQPTTJCMFDBVTFTGPS GBJMVSFTUIBUBQQFBSJEFOUJDBMUPUIFVTFS ɾ7.*NBHFNJTDPOGJHVSBUJPO ɾ"1*4FSWFS%PXO ɾ#BDLFOE4UPSBHF%PXO ɾ"VUIFOUJDBUJPOTFSWFSEPXO ɾ4DIFEVMFSNJTDPOGJHVSBUJPO ɾ.JEEMFXBSFTFSWFSEPXO ɾFUD

Slide 12

Slide 12 text

%JGGJDVMUJFTJOUSPVCMFTIPPUJOH ɾ3PVUF"EWFSUJTF ɾ/FUXPSLJOHJO)7 ɾ%)$1GPS7. ɾ4FOESFDFJWF31$T l*DBOOPUDSFBUFOFX7.z "OEXFOFFEUPCFGBNJMJBSXJUIBXJEFMBZFSPG UFDIOPMPHZUPCFBCMFUPEFBMXJUIJUQSPQFSMZ ɾ7.*NBHFNJTDPOGJHVSBUJPO ɾ"1*4FSWFS%PXO ɾ#BDLFOE4UPSBHF%PXO ɾ"VUIFOUJDBUJPOTFSWFSEPXO ɾ4DIFEVMFSNJTDPOGJHVSBUJPO ɾ.JEEMFXBSFTFSWFSEPXO ɾFUD

Slide 13

Slide 13 text

7FSEBIBTNBOZTFSWJDFTʜ )FSPLV Elasticsearch

Slide 14

Slide 14 text

ݱঢ়ͷ՝୊

Slide 15

Slide 15 text

7FSEBIBTNBOZTFSWJDFTʜ )FSPLV Elasticsearch

Slide 16

Slide 16 text

7FSEBIBTNBOZTFSWJDFTʜ %VFUPEJGGFSFODFTJOEFWFMPQNFOUUJNF BOEDVMUVSBMJTTVFT • &BDITFSWJDFIBTBEJGGFSFOUEFQMPZNFOUNFUIPE • 0UIFSTFSWJDFEFWFMPQFSTSBSFMZLOPXXIBULJOEPG UFTUJOHBTFSWJDFJTEPJOH • 5IFZNBZOPULOPXUIBUPUIFSTFSWJDFTEFQFOEPO UIFJSTFSWJDFT

Slide 17

Slide 17 text

.BOVBMMZDPOUSPMEFQMPZNFOU BOETJEFFGGFDUT 8FBSFEPJOHUIFGPMMPXJOHUIJOHTNBOVBMMZ ɾ$IFDL UIF JNQBDU PG UIF DIBOHF PO PUIFS TFSWJDFT ɾ$IFDL UIF JNQBDU PG UIBU EFQMPZNFOU PO UIF VTFS ɾ2" BSSBOHFNFOUT ɾ1SPHSFTT NBOBHFNFOU ʜ.PTUPGUIFUJNF JUTOPUEPOFXFMMFOPVHI

Slide 18

Slide 18 text

%FMBZPGGBJMVSFEFUFDUJPO *GUIFJNQBDUDIFDLJTNJTTFE PSJGUIFEFQMPZNFOUGBJMTʜ 5IFQFSTPOJODIBSHFPGEFQMPZNFOUIBTOPXBZ PGLOPXJOHUIFJNQBDUJNNFEJBUFMZ "GGFDUFETFSWJDFTNBZCFEPXOEVFUPGBJMVSFT UIBUBSFOPUJOUIFJSBSFBPGSFTQPOTJCJMJUZ

Slide 19

Slide 19 text

σϓϩΠʹ૊ࠐΈ͍ͨػೳͱ ͦΕʹΑͬͯղܾͰ͖Δ͜ͱ

Slide 20

Slide 20 text

&OGPSDJOHBSPMMCBDLGSJFOEMZEFQMPZ *UJTJNQPSUBOUUPCFBCMFUPRVJDLMZSPMMCBDL UIFTUBUFPGTFSWJDFJOUIFBDDJEFOU %FQMPZJOH"QQUIBUDPOUBJOEFQFOEFODJFTTVDI BTMJCSBSJFTDPOUBJOFSJ[F %FQMPZXJUIBTFUPGFYUFSOBMJOGPSNBUJPOTVDI BTDPOGJHTBWFUIFTFUPO(JUSFQP

Slide 21

Slide 21 text

%FQFOEFODZCBTFEIBOEMFS 'VODUJPOUPVTFTFSWJDFEFQFOEFODZNBQUP BTTJTUXJUIQSPQFSOPUJGJDBUJPOBOESPMMCBDL 3FDFJWFEPXOOPUJGJDBUJPOBCPVUBOZTFSWJDFBOE JTTVFOPUJGJDBUJPOUPEFWFMPQFSTPGTFSWJDFTUIBU EFQFOEPOUIBU "VUPNBUJDBMMZOPUJGZEFWFMPQFSTXIPNBOBHF TFSWJDFUIBUEFQFOETFSWJDFEFQMPZNFOU

Slide 22

Slide 22 text

"VUPNBUJDNFUSJDTCBTFEPQFSBUJPO 'VODUJPOUIBUXBSOTBHBJOTUPSBVUPNBUJDBMMZ SPMMCBDLTVEEFOJODSFBTFTJOSFTPVSDFVTBHF PSPUIFSNFUSJDTCFGPSFBOEBGUFSEFQMPZNFOU .POJUPSJOHBOEBDUJPOPOUIFTFSWJDFTPXO DSJUJDBMNFUSJDT .POJUPSJOHBOEOPUJGJDBUJPODSJUJDBMNFUSJDTPG UIFTFSWJDFDPVOUJFTUIBUEFQFOEPOUIBUTFSWJDF

Slide 23

Slide 23 text

1PTUEFQMPZNFOUBOEQFSJPEJDUFTU "VUPNBUJDBMMZFYFDUFTUTPOEFQMPZFETFSWJDF BOEGPMMPXTFSWJDFT 4FMFDUJPOPGTFSWJDFTUPCFUFTUFECBTFEPO EFQFOEFODZNBQBOEUFTUFYFDVUJPO %FUFDUJPOPGVOFYQFDUFEGBJMVSFTUISPVHI QFSJPEJDUFTUFYFDVUJPO

Slide 24

Slide 24 text

LTͷ࢓૊ΈΛ࢖ͬͨ ࣮૷ͷΞΠσΞ

Slide 25

Slide 25 text

0WFSWJFX Github (on-premise) App ArgoCD Kubewatch Tester Notifier Dependency Mapper Slack/ Alertmanager

Slide 26

Slide 26 text

LT"SHP$%,VCFXBUDI "QQMZUIFNBOJGFTUPOUIF(JUSFQP .BOBHFNFOU/PUJGJDBUJPO 3VOXJUIUIFNFDIBOJTNUPFYFDVUFEFQMPZNFOU CBTFEPOBQPTUPO(JUIVC3FMFBTF 0QFSBUFBXFCIPPLCBTFEOPUJGJDBUJPOTFSWJDFUP SFDFJWFBOEQSPQFSMZOPUJGZUIFSFTVMUTPG SFTPVSDFNPOJUPSJOHCZ,VCFXBUDIBOEUFTUJOH

Slide 27

Slide 27 text

LT"SHP$%,VCFXBUDI "QQMZUIFNBOJGFTUPOUIF(JUSFQP .BOBHFNFOU/PUJGJDBUJPO &OGPSDFNFUBEBUBTFUUJOHTXJUI"ENJTTJPO $POUSPM

Slide 28

Slide 28 text

5FTUTZTUFNCBTFEPO3FDPODJMFMPPQ 0QFSBUPSNBOBHFUFTUFYFDBOESFTVMUTGSPN SFTPVSDFDIBOHFBOEQFSJPEJDFWFOUT .BJOUBJOBMJTUPGKPCNBOJGFTUTGPSUFTUJOHBTB DVTUPNSFTPVSDFBTTPDJBUFEXJUIUIFTFSWJDF $POUSPMFYFDVUJPOPGUFTUTGPSSFMBUFETFSWJDFT CBTFEPOUIFEFQFOEFODZNBQXIFOBOPUIFS TFSWJDFEFQMPZNFOUFWFOUPDDVST

Slide 29

Slide 29 text

.FUSJDTCBTFEOPUJGJDBUJPOTZTUFN .POJUPSTQSFEFUFSNJOFELVCFTUBUFNFUSJDT JOVOJUTPG%FQMPZNFOUBOEOPUJGJFT CBTFEPOTVEEFODIBOHFT *TTVFBMFSUGSPN"MFSUNBOBHFSUPUIF BGPSFNFOUJPOFEXFCIPPLCBTFEOPUJGJDBUJPOTZTUFN 0GDPVSTF OPUJGJDBUJPOXJMMCFFYFDVUFECBTFEPO UIFEFQFOEFODZNBQ

Slide 30

Slide 30 text

0WFSWJFX Github (on-premise) App ArgoCD Kubewatch Tester Notifier Dependency Mapper Slack/ Alertmanager

Slide 31

Slide 31 text

7FSEBͰͷ ཁૉٕज़ͷ ಋೖࣄྫ

Slide 32

Slide 32 text

0QFSBUJPOXJUI3FDPODJMF-PPQ .BOBHFUIFTUBUFPG/'7JOTUBODF BTLTSFTPVSDF • 3PVUFS 71/(BUFXBZ FUDDBOCFPQFSBUFEBT QVSFLTSFTPVSDFT • &GGPSUTUPEFGJOFLTBTUIFGPVOEBUJPOGPS FYFDVUJOHTUSVDUVSFEEBUBTUPSFTBOEBDUJPOT SFMBUFEUPUIBUEBUB

Slide 33

Slide 33 text

No content

Slide 34

Slide 34 text

0QFSBUJPOXJUI3FDPODJMF-PPQ .BOBHFUIFTUBUFPG/'7JOTUBODF BTLTSFTPVSDF https://www.janog.gr.jp/meeting/janog48/wp- content/uploads/2021/07/janog48-LINENFV- %E5%9F%8E%E5%80%89%E5%BC%98%E6%A8%B9.pdf

Slide 35

Slide 35 text

%QMBOFEFQMPZNFOULTMJ[BUJPO &OBCMJOHEBUBQMBOFTUIBUNVTUCFEFQMPZFE EJSFDUMZUPCBSFNFUBMUPCFEFQMPZFEGSPN LT • .BJOUBSHFU)ZQFSWJTPSTPG0QFO4UBDL • $VSSFOUMZ UIFGPDVTJTPOEFDMBSBUJWF EFQMPZNFOUBOEQSPQFSGPSLJOHPG"OTJCMF FYFDVUJPO

Slide 36

Slide 36 text

%QMBOFEFQMPZNFOULTMJ[BUJPO &OBCMJOHEBUBQMBOFTUIBUNVTUCFEFQMPZFE EJSFDUMZUPCBSFNFUBMUPCFEFQMPZFEGSPN LT https://engineering.linecorp.com/ja/blog/internship- report20-hypervisor-update-system-with-kubernetes/

Slide 37

Slide 37 text

6TF"SHP$% 6TJOH"SHP$%GPS,BB4"1*EFQMPZNFOU • &GGJDJFOUQSFQBSBUJPOVTJOHBXFBMUIPGSFMBUFE LTSFMBUFEUPPMT • *ODSFBTJOHUIFWJTJCJMJUZBOETQFFEPGQSPHSFTT CZNPWJOHBXBZGSPNBQSPDFEVSBMBQQSPBDI

Slide 38

Slide 38 text

·ͱΊ

Slide 39

Slide 39 text

%FQMPZNFOUCZEFDMBSBUJPO • %FQMPZNFOUUBSHFUTDBOCFBCTUSBDUFEBOE NBOBHFEBTLTSFTPVSDFTUPFOGPSDFUIF BTTJHONFOUPGNFUBEBUB SFTPVSDFMJNJUT FUD • 3FQSFTFOUJOHEFQMPZNFOUBTBDIBJOPG QSPDFTTFTXJUITNBMMSFTQPOTJCJMJUJFTCBTFEPO SFTPVSDFT SBUIFSUIBOBTFSJFTPGQSPDFEVSFT NBLFTJUFBTJFSUPNBOBHFSPMMCBDLBOEPUIFS TDFOBSJPT

Slide 40

Slide 40 text

1PTUEFQMPZNFOU UFTUT CZEFQFOEFODZNBQ • *OBEEJUJPOUPUIFIFBMUIPGUIFTFSWJDF JUBMTP OFFETUPCFDIFDLFEJGTPNFSPVUJOFBENJO PQFSBUJPOTBSFQPTTJCMF • #BTFEPOUIFEFQFOEFODZNBQ SVOFWFOUCBTFE UFTUTPOPUIFSTFSWJDFTUIBUEFQFOEPOUIF TFSWJDFUPCFEFQMPZFE

Slide 41

Slide 41 text

%FQMPZNFOUOPUJGJDBUJPO CZEFQFOEFODZNBQ • /PUJGZSFMBUFETFSWJDFTPGEFQMPZNFOUGBJMVSFT UFTUGBJMVSFT FUDVTJOHEFQFOEFODZNBQTUP TUSFBNMJOFJNQBDUDIFDLT • *UDBOCFBHFOFSJDGSBNFXPSLGPSSPVUJOFIFBMUI DIFDLT FTQFDJBMMZGPSFWFOUCBTFEBOEQFSJPEJD OPUJGJDBUJPOPGUFTUSFTVMUT

Slide 42

Slide 42 text

5IFQSFNJTFJTʜ • :PVTIPVMEDIFDLUIFJNQMFNFOUBUJPOPGUIF TFSWJDF)PXTIPVMEJUEFQFOEPOPUIFS TFSWJDFT • 8IBULJOEPGDIFDLTTIPVMECFQFSGPSNFEJO TUBHJOHBOEVOJUUFTUTUPQSFWFOUQSPCMFNTJO QSPEVDUJPOEFQMPZNFOUT 1FSGFDUEFQMPZNFOUJTOPUBTJMWFSCVMMFU

Slide 43

Slide 43 text

THANK YOU