LINEのPrivate Cloudを支えるデプロイ基盤の課題解決に向けた取り組み / Design of deployment platform for resolving operational issues of LINE's Private Cloud

LINEのPrivate Cloudを⽀えるデプロイ基盤の課題解決に向けた取り組み Wataru Manji – LINE Corp. 2021.09

LINEのPrivate Cloudに対するデプロイ基盤による課題解決 Wataru Manji – LINE Corp. 2021.09

஫ɿ·ͩ΍ͬͯͳ͍

ొஃऀ঺հ /55ιϑτ΢ΣΞ ݱ /55ςΫϊΫϩε ೖࣾ ɾ0QFO4UBDLͷݕূɺࣾ಺Ϋϥ΢υ։ൃ ɾ/8ػثͷΦϖϨʔγϣϯޮ཰Խͷݚڀ ɾ/55άϧʔϓͷٕज़ΠϕϯτӡӦ -*/&ೖࣾ 7FSEB3FMJBCJMJUZ&OHJOFFSJOH
ɾ؂ࢹํࣜͷݟ௚͠ɺΞϥʔτ੔ཧɺΦϯίʔϧͷಋೖ ɾσϓϩΠํࣜͷվળ ɾͦͷଞϓϩδΣΫτاըͳͲ dϚωʔδϟʔ झຯ͸ےτϨͱࢄา ᤈ࣏ ব .BOBHFS 7FSEB 3FMJBCJMJUZ &OHJOFFSJOH 5FBN !@NBOKJ

7FSEBͷ঺հ αʔϏεͷෳࡶੑʹ͍ͭͯ ݱঢ়ͷ՝୊ σϓϩΠʹ૊ࠐΈ͍ͨػೳͱͦΕʹΑͬͯղܾͰ͖Δ͜ͱ LTͷ࢓૊ΈΛ࢖࣮ͬͨ૷ͷΞΠσΞ 7FSEBͰͷ ཁૉٕज़ͷ ಋೖࣄྫ まとめ 01
02 03 04 05 06 07 Contents

7FSEBͷ঺հ

8IBUJT7FSEB

5IFTFSWJDFTPG7FSEB )FSPLV Elasticsearch

αʔϏεͷෳࡶੑʹ͍ͭͯ

4FSWJDFEFQFOEFODJFT 4PNF7FSEBTFSWJDFTBSFEFQFOEFOUPOPUIFS7FSEBTFSWJDFT

%JGGJDVMUJFTJOUSPVCMFTIPPUJOH ɾ3PVUF"EWFSUJTF ɾ/FUXPSLJOHJO)7 ɾ%)$1GPS7. ɾ4FOESFDFJWF31$T l*DBOOPUDSFBUFOFX7.z ʜCVU5IFSFBSFNVMUJQMFQPTTJCMFDBVTFTGPS GBJMVSFTUIBUBQQFBSJEFOUJDBMUPUIFVTFS ɾ7.*NBHFNJTDPOGJHVSBUJPO ɾ"1*4FSWFS%PXO
ɾ#BDLFOE4UPSBHF%PXO ɾ"VUIFOUJDBUJPOTFSWFSEPXO ɾ4DIFEVMFSNJTDPOGJHVSBUJPO ɾ.JEEMFXBSFTFSWFSEPXO ɾFUD

%JGGJDVMUJFTJOUSPVCMFTIPPUJOH ɾ3PVUF"EWFSUJTF ɾ/FUXPSLJOHJO)7 ɾ%)$1GPS7. ɾ4FOESFDFJWF31$T l*DBOOPUDSFBUFOFX7.z "OEXFOFFEUPCFGBNJMJBSXJUIBXJEFMBZFSPG UFDIOPMPHZUPCFBCMFUPEFBMXJUIJUQSPQFSMZ ɾ7.*NBHFNJTDPOGJHVSBUJPO ɾ"1*4FSWFS%PXO
ɾ#BDLFOE4UPSBHF%PXO ɾ"VUIFOUJDBUJPOTFSWFSEPXO ɾ4DIFEVMFSNJTDPOGJHVSBUJPO ɾ.JEEMFXBSFTFSWFSEPXO ɾFUD

7FSEBIBTNBOZTFSWJDFTʜ )FSPLV Elasticsearch

ݱঢ়ͷ՝୊

7FSEBIBTNBOZTFSWJDFTʜ )FSPLV Elasticsearch

7FSEBIBTNBOZTFSWJDFTʜ %VFUPEJGGFSFODFTJOEFWFMPQNFOUUJNF BOEDVMUVSBMJTTVFT • &BDITFSWJDFIBTBEJGGFSFOUEFQMPZNFOUNFUIPE • 0UIFSTFSWJDFEFWFMPQFSTSBSFMZLOPXXIBULJOEPG UFTUJOHBTFSWJDFJTEPJOH • 5IFZNBZOPULOPXUIBUPUIFSTFSWJDFTEFQFOEPO
UIFJSTFSWJDFT

.BOVBMMZDPOUSPMEFQMPZNFOU BOETJEFFGGFDUT 8FBSFEPJOHUIFGPMMPXJOHUIJOHTNBOVBMMZ ɾ$IFDL UIF JNQBDU PG UIF DIBOHF PO
PUIFS TFSWJDFT ɾ$IFDL UIF JNQBDU PG UIBU EFQMPZNFOU PO UIF VTFS ɾ2" BSSBOHFNFOUT ɾ1SPHSFTT NBOBHFNFOU ʜ.PTUPGUIFUJNF JUTOPUEPOFXFMMFOPVHI

%FMBZPGGBJMVSFEFUFDUJPO *GUIFJNQBDUDIFDLJTNJTTFE PSJGUIFEFQMPZNFOUGBJMTʜ 5IFQFSTPOJODIBSHFPGEFQMPZNFOUIBTOPXBZ PGLOPXJOHUIFJNQBDUJNNFEJBUFMZ "GGFDUFETFSWJDFTNBZCFEPXOEVFUPGBJMVSFT UIBUBSFOPUJOUIFJSBSFBPGSFTQPOTJCJMJUZ

σϓϩΠʹ૊ࠐΈ͍ͨػೳͱ ͦΕʹΑͬͯղܾͰ͖Δ͜ͱ

&OGPSDJOHBSPMMCBDLGSJFOEMZEFQMPZ *UJTJNQPSUBOUUPCFBCMFUPRVJDLMZSPMMCBDL UIFTUBUFPGTFSWJDFJOUIFBDDJEFOU %FQMPZJOH"QQUIBUDPOUBJOEFQFOEFODJFTTVDI BTMJCSBSJFTDPOUBJOFSJ[F %FQMPZXJUIBTFUPGFYUFSOBMJOGPSNBUJPOTVDI BTDPOGJHTBWFUIFTFUPO(JUSFQP

%FQFOEFODZCBTFEIBOEMFS 'VODUJPOUPVTFTFSWJDFEFQFOEFODZNBQUP BTTJTUXJUIQSPQFSOPUJGJDBUJPOBOESPMMCBDL 3FDFJWFEPXOOPUJGJDBUJPOBCPVUBOZTFSWJDFBOE JTTVFOPUJGJDBUJPOUPEFWFMPQFSTPGTFSWJDFTUIBU EFQFOEPOUIBU "VUPNBUJDBMMZOPUJGZEFWFMPQFSTXIPNBOBHF TFSWJDFUIBUEFQFOETFSWJDFEFQMPZNFOU

"VUPNBUJDNFUSJDTCBTFEPQFSBUJPO 'VODUJPOUIBUXBSOTBHBJOTUPSBVUPNBUJDBMMZ SPMMCBDLTVEEFOJODSFBTFTJOSFTPVSDFVTBHF PSPUIFSNFUSJDTCFGPSFBOEBGUFSEFQMPZNFOU .POJUPSJOHBOEBDUJPOPOUIFTFSWJDFTPXO DSJUJDBMNFUSJDT .POJUPSJOHBOEOPUJGJDBUJPODSJUJDBMNFUSJDTPG UIFTFSWJDFDPVOUJFTUIBUEFQFOEPOUIBUTFSWJDF

1PTUEFQMPZNFOUBOEQFSJPEJDUFTU "VUPNBUJDBMMZFYFDUFTUTPOEFQMPZFETFSWJDF BOEGPMMPXTFSWJDFT 4FMFDUJPOPGTFSWJDFTUPCFUFTUFECBTFEPO EFQFOEFODZNBQBOEUFTUFYFDVUJPO %FUFDUJPOPGVOFYQFDUFEGBJMVSFTUISPVHI QFSJPEJDUFTUFYFDVUJPO

LTͷ࢓૊ΈΛ࢖ͬͨ ࣮૷ͷΞΠσΞ

0WFSWJFX Github (on-premise) App ArgoCD Kubewatch Tester Notifier Dependency Mapper
Slack/ Alertmanager

LT "SHP$% ,VCFXBUDI "QQMZUIFNBOJGFTUPOUIF(JUSFQP .BOBHFNFOU/PUJGJDBUJPO 3VOXJUIUIFNFDIBOJTNUPFYFDVUFEFQMPZNFOU CBTFEPOBQPTUPO(JUIVC3FMFBTF 0QFSBUFBXFCIPPLCBTFEOPUJGJDBUJPOTFSWJDFUP
SFDFJWFBOEQSPQFSMZOPUJGZUIFSFTVMUTPG SFTPVSDFNPOJUPSJOHCZ,VCFXBUDIBOEUFTUJOH

LT "SHP$% ,VCFXBUDI "QQMZUIFNBOJGFTUPOUIF(JUSFQP .BOBHFNFOU/PUJGJDBUJPO &OGPSDFNFUBEBUBTFUUJOHTXJUI"ENJTTJPO $POUSPM

5FTUTZTUFNCBTFEPO3FDPODJMFMPPQ 0QFSBUPSNBOBHFUFTUFYFDBOESFTVMUTGSPN SFTPVSDFDIBOHFBOEQFSJPEJDFWFOUT .BJOUBJOBMJTUPGKPCNBOJGFTUTGPSUFTUJOHBTB DVTUPNSFTPVSDFBTTPDJBUFEXJUIUIFTFSWJDF $POUSPMFYFDVUJPOPGUFTUTGPSSFMBUFETFSWJDFT CBTFEPOUIFEFQFOEFODZNBQXIFOBOPUIFS TFSWJDFEFQMPZNFOUFWFOUPDDVST

.FUSJDTCBTFEOPUJGJDBUJPOTZTUFN .POJUPSTQSFEFUFSNJOFELVCFTUBUFNFUSJDT JOVOJUTPG%FQMPZNFOUBOEOPUJGJFT CBTFEPOTVEEFODIBOHFT *TTVFBMFSUGSPN"MFSUNBOBHFSUPUIF BGPSFNFOUJPOFEXFCIPPLCBTFEOPUJGJDBUJPOTZTUFN 0GDPVSTF OPUJGJDBUJPOXJMMCFFYFDVUFECBTFEPO
UIFEFQFOEFODZNBQ

0WFSWJFX Github (on-premise) App ArgoCD Kubewatch Tester Notifier Dependency Mapper
Slack/ Alertmanager

7FSEBͰͷ ཁૉٕज़ͷ ಋೖࣄྫ

0QFSBUJPOXJUI3FDPODJMF-PPQ .BOBHFUIFTUBUFPG/'7JOTUBODF BTLTSFTPVSDF • 3PVUFS 71/(BUFXBZ FUDDBOCFPQFSBUFEBT QVSFLTSFTPVSDFT • &GGPSUTUPEFGJOFLTBTUIFGPVOEBUJPOGPS
FYFDVUJOHTUSVDUVSFEEBUBTUPSFTBOEBDUJPOT SFMBUFEUPUIBUEBUB

0QFSBUJPOXJUI3FDPODJMF-PPQ .BOBHFUIFTUBUFPG/'7JOTUBODF BTLTSFTPVSDF https://www.janog.gr.jp/meeting/janog48/wp- content/uploads/2021/07/janog48-LINENFV- %E5%9F%8E%E5%80%89%E5%BC%98%E6%A8%B9.pdf

%QMBOFEFQMPZNFOULTMJ[BUJPO &OBCMJOHEBUBQMBOFTUIBUNVTUCFEFQMPZFE EJSFDUMZUPCBSFNFUBMUPCFEFQMPZFEGSPN LT • .BJOUBSHFU)ZQFSWJTPSTPG0QFO4UBDL • $VSSFOUMZ UIFGPDVTJTPOEFDMBSBUJWF EFQMPZNFOUBOEQSPQFSGPSLJOHPG"OTJCMF
FYFDVUJPO

%QMBOFEFQMPZNFOULTMJ[BUJPO &OBCMJOHEBUBQMBOFTUIBUNVTUCFEFQMPZFE EJSFDUMZUPCBSFNFUBMUPCFEFQMPZFEGSPN LT https://engineering.linecorp.com/ja/blog/internship- report20-hypervisor-update-system-with-kubernetes/

6TF"SHP$% 6TJOH"SHP$%GPS,BB4"1*EFQMPZNFOU • &GGJDJFOUQSFQBSBUJPOVTJOHBXFBMUIPGSFMBUFE LTSFMBUFEUPPMT • *ODSFBTJOHUIFWJTJCJMJUZBOETQFFEPGQSPHSFTT CZNPWJOHBXBZGSPNBQSPDFEVSBMBQQSPBDI

·ͱΊ

%FQMPZNFOUCZEFDMBSBUJPO • %FQMPZNFOUUBSHFUTDBOCFBCTUSBDUFEBOE NBOBHFEBTLTSFTPVSDFTUPFOGPSDFUIF BTTJHONFOUPGNFUBEBUB SFTPVSDFMJNJUT FUD • 3FQSFTFOUJOHEFQMPZNFOUBTBDIBJOPG QSPDFTTFTXJUITNBMMSFTQPOTJCJMJUJFTCBTFEPO
SFTPVSDFT SBUIFSUIBOBTFSJFTPGQSPDFEVSFT NBLFTJUFBTJFSUPNBOBHFSPMMCBDLBOEPUIFS TDFOBSJPT

1PTUEFQMPZNFOU UFTUT CZEFQFOEFODZNBQ • *OBEEJUJPOUPUIFIFBMUIPGUIFTFSWJDF JUBMTP OFFETUPCFDIFDLFEJGTPNFSPVUJOFBENJO PQFSBUJPOTBSFQPTTJCMF • #BTFEPOUIFEFQFOEFODZNBQ
SVOFWFOUCBTFE UFTUTPOPUIFSTFSWJDFTUIBUEFQFOEPOUIF TFSWJDFUPCFEFQMPZFE

%FQMPZNFOUOPUJGJDBUJPO CZEFQFOEFODZNBQ • /PUJGZSFMBUFETFSWJDFTPGEFQMPZNFOUGBJMVSFT UFTUGBJMVSFT FUDVTJOHEFQFOEFODZNBQTUP TUSFBNMJOFJNQBDUDIFDLT • *UDBOCFBHFOFSJDGSBNFXPSLGPSSPVUJOFIFBMUI
DIFDLT FTQFDJBMMZGPSFWFOUCBTFEBOEQFSJPEJD OPUJGJDBUJPOPGUFTUSFTVMUT

5IFQSFNJTFJTʜ • :PVTIPVMEDIFDLUIFJNQMFNFOUBUJPOPGUIF TFSWJDF)PXTIPVMEJUEFQFOEPOPUIFS TFSWJDFT • 8IBULJOEPGDIFDLTTIPVMECFQFSGPSNFEJO TUBHJOHBOEVOJUUFTUTUPQSFWFOUQSPCMFNTJO QSPEVDUJPOEFQMPZNFOUT 1FSGFDUEFQMPZNFOUJTOPUBTJMWFSCVMMFU

THANK YOU

LINEのPrivate Cloudを支える デプロイ基盤の課題解決に向けた取り組み / De...

LINEのPrivate Cloudを支える デプロイ基盤の課題解決に向けた取り組み / Design of deployment platform for resolving operational issues of LINE's Private Cloud

More Decks by LINE Developers

Other Decks in Technology

Featured

Transcript

LINEのPrivate Cloudを支えるデプロイ基盤の課題解決に向けた取り組み / De...

LINEのPrivate Cloudを支えるデプロイ基盤の課題解決に向けた取り組み / Design of deployment platform for resolving operational issues of LINE's Private Cloud