Upgrade to Pro — share decks privately, control downloads, hide ads and more …

LINEのPrivate Cloudを支える デプロイ基盤の課題解決に向けた取り組み / Design of deployment platform for resolving operational issues of LINE's Private Cloud

LINEのPrivate Cloudを支える デプロイ基盤の課題解決に向けた取り組み / Design of deployment platform for resolving operational issues of LINE's Private Cloud

CI/CD Conference 2021 by CloudNative Days での登壇資料です
https://event.cloudnativedays.jp/cicd2021

私達が開発しているLINEのPrivate Cloud PlatformであるVerdaは、社内開発者向けに多種多様なインフラサービスを管理してます。そして、あるサービスが別のVerdaサービスに依存していることも多く、大局的に見るとデザインの中に大量の疎結合部が存在する複雑性の高いMSAとして稼動しています。
そのような前提の元、最近はサービス規模と提供サービスの種類の拡大に応じてデザインの複雑度やトラブル時の影響度が上がってきているのですが、それぞれのサービスのデプロイプロセスはその状況変化に追従できておらず、デプロイの管理コストが大きくなっていることが課題として見えてきました。
このセッションではVerdaのデプロイについて簡単に歴史の振り返りと課題の明確化を行ない、それをどのように解決するべきかのアイデアといくつかの実例についてご紹介します。

萬治 渉
LINE株式会社
Manager, Verda Reliability Engineering Team

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers
PRO

September 03, 2021
Tweet

Transcript

  1. LINEのPrivate Cloudを⽀える デプロイ基盤の課題解決に向けた取り組み Wataru Manji – LINE Corp. 2021.09

  2. LINEのPrivate Cloudに対する デプロイ基盤による課題解決 Wataru Manji – LINE Corp. 2021.09

  3. ஫ɿ·ͩ΍ͬͯͳ͍

  4. ొஃऀ঺հ /55ιϑτ΢ΣΞ ݱ /55ςΫϊΫϩε ೖࣾ ɾ0QFO4UBDLͷݕূɺࣾ಺Ϋϥ΢υ։ൃ ɾ/8ػثͷΦϖϨʔγϣϯޮ཰Խͷݚڀ ɾ/55άϧʔϓͷٕज़ΠϕϯτӡӦ -*/&ೖࣾ 7FSEB3FMJBCJMJUZ&OHJOFFSJOH

    ɾ؂ࢹํࣜͷݟ௚͠ɺΞϥʔτ੔ཧɺΦϯίʔϧͷಋೖ ɾσϓϩΠํࣜͷվળ ɾͦͷଞϓϩδΣΫτاըͳͲ dϚωʔδϟʔ झຯ͸ےτϨͱࢄา ᤈ࣏ ব .BOBHFS 7FSEB 3FMJBCJMJUZ &OHJOFFSJOH 5FBN !@NBOKJ
  5. 7FSEBͷ঺հ αʔϏεͷෳࡶੑʹ͍ͭͯ ݱঢ়ͷ՝୊ σϓϩΠʹ૊ࠐΈ͍ͨػೳͱͦΕʹΑͬͯղܾͰ͖Δ͜ͱ LTͷ࢓૊ΈΛ࢖࣮ͬͨ૷ͷΞΠσΞ 7FSEBͰͷ ཁૉٕज़ͷ ಋೖࣄྫ まとめ 01

    02 03 04 05 06 07 Contents
  6. 7FSEBͷ঺հ

  7. 8IBUJT7FSEB

  8. 5IFTFSWJDFTPG7FSEB )FSPLV Elasticsearch

  9. αʔϏεͷෳࡶੑʹ͍ͭͯ

  10. 4FSWJDFEFQFOEFODJFT 4PNF7FSEBTFSWJDFTBSFEFQFOEFOUPOPUIFS7FSEBTFSWJDFT

  11. %JGGJDVMUJFTJOUSPVCMFTIPPUJOH ɾ3PVUF"EWFSUJTF ɾ/FUXPSLJOHJO)7 ɾ%)$1GPS7. ɾ4FOESFDFJWF31$T l*DBOOPUDSFBUFOFX7.z ʜCVU5IFSFBSFNVMUJQMFQPTTJCMFDBVTFTGPS GBJMVSFTUIBUBQQFBSJEFOUJDBMUPUIFVTFS ɾ7.*NBHFNJTDPOGJHVSBUJPO ɾ"1*4FSWFS%PXO

    ɾ#BDLFOE4UPSBHF%PXO ɾ"VUIFOUJDBUJPOTFSWFSEPXO ɾ4DIFEVMFSNJTDPOGJHVSBUJPO ɾ.JEEMFXBSFTFSWFSEPXO ɾFUD
  12. %JGGJDVMUJFTJOUSPVCMFTIPPUJOH ɾ3PVUF"EWFSUJTF ɾ/FUXPSLJOHJO)7 ɾ%)$1GPS7. ɾ4FOESFDFJWF31$T l*DBOOPUDSFBUFOFX7.z "OEXFOFFEUPCFGBNJMJBSXJUIBXJEFMBZFSPG UFDIOPMPHZUPCFBCMFUPEFBMXJUIJUQSPQFSMZ ɾ7.*NBHFNJTDPOGJHVSBUJPO ɾ"1*4FSWFS%PXO

    ɾ#BDLFOE4UPSBHF%PXO ɾ"VUIFOUJDBUJPOTFSWFSEPXO ɾ4DIFEVMFSNJTDPOGJHVSBUJPO ɾ.JEEMFXBSFTFSWFSEPXO ɾFUD
  13. 7FSEBIBTNBOZTFSWJDFTʜ )FSPLV Elasticsearch

  14. ݱঢ়ͷ՝୊

  15. 7FSEBIBTNBOZTFSWJDFTʜ )FSPLV Elasticsearch

  16. 7FSEBIBTNBOZTFSWJDFTʜ %VFUPEJGGFSFODFTJOEFWFMPQNFOUUJNF BOEDVMUVSBMJTTVFT • &BDITFSWJDFIBTBEJGGFSFOUEFQMPZNFOUNFUIPE • 0UIFSTFSWJDFEFWFMPQFSTSBSFMZLOPXXIBULJOEPG UFTUJOHBTFSWJDFJTEPJOH • 5IFZNBZOPULOPXUIBUPUIFSTFSWJDFTEFQFOEPO

    UIFJSTFSWJDFT
  17. .BOVBMMZDPOUSPMEFQMPZNFOU BOETJEFFGGFDUT 8FBSFEPJOHUIFGPMMPXJOHUIJOHTNBOVBMMZ ɾ$IFDL UIF JNQBDU PG UIF DIBOHF PO

    PUIFS TFSWJDFT ɾ$IFDL UIF JNQBDU PG UIBU EFQMPZNFOU PO UIF VTFS ɾ2" BSSBOHFNFOUT ɾ1SPHSFTT NBOBHFNFOU ʜ.PTUPGUIFUJNF JUTOPUEPOFXFMMFOPVHI
  18. %FMBZPGGBJMVSFEFUFDUJPO *GUIFJNQBDUDIFDLJTNJTTFE  PSJGUIFEFQMPZNFOUGBJMTʜ  5IFQFSTPOJODIBSHFPGEFQMPZNFOUIBTOPXBZ PGLOPXJOHUIFJNQBDUJNNFEJBUFMZ  "GGFDUFETFSWJDFTNBZCFEPXOEVFUPGBJMVSFT UIBUBSFOPUJOUIFJSBSFBPGSFTQPOTJCJMJUZ

  19. σϓϩΠʹ૊ࠐΈ͍ͨػೳͱ ͦΕʹΑͬͯղܾͰ͖Δ͜ͱ

  20. &OGPSDJOHBSPMMCBDLGSJFOEMZEFQMPZ *UJTJNQPSUBOUUPCFBCMFUPRVJDLMZSPMMCBDL UIFTUBUFPGTFSWJDFJOUIFBDDJEFOU  %FQMPZJOH"QQUIBUDPOUBJOEFQFOEFODJFTTVDI BTMJCSBSJFTDPOUBJOFSJ[F  %FQMPZXJUIBTFUPGFYUFSOBMJOGPSNBUJPOTVDI BTDPOGJHTBWFUIFTFUPO(JUSFQP

  21. %FQFOEFODZCBTFEIBOEMFS 'VODUJPOUPVTFTFSWJDFEFQFOEFODZNBQUP BTTJTUXJUIQSPQFSOPUJGJDBUJPOBOESPMMCBDL  3FDFJWFEPXOOPUJGJDBUJPOBCPVUBOZTFSWJDFBOE JTTVFOPUJGJDBUJPOUPEFWFMPQFSTPGTFSWJDFTUIBU EFQFOEPOUIBU  "VUPNBUJDBMMZOPUJGZEFWFMPQFSTXIPNBOBHF TFSWJDFUIBUEFQFOETFSWJDFEFQMPZNFOU

  22. "VUPNBUJDNFUSJDTCBTFEPQFSBUJPO 'VODUJPOUIBUXBSOTBHBJOTUPSBVUPNBUJDBMMZ SPMMCBDLTVEEFOJODSFBTFTJOSFTPVSDFVTBHF PSPUIFSNFUSJDTCFGPSFBOEBGUFSEFQMPZNFOU  .POJUPSJOHBOEBDUJPOPOUIFTFSWJDFTPXO DSJUJDBMNFUSJDT  .POJUPSJOHBOEOPUJGJDBUJPODSJUJDBMNFUSJDTPG UIFTFSWJDFDPVOUJFTUIBUEFQFOEPOUIBUTFSWJDF

  23. 1PTUEFQMPZNFOUBOEQFSJPEJDUFTU "VUPNBUJDBMMZFYFDUFTUTPOEFQMPZFETFSWJDF BOEGPMMPXTFSWJDFT  4FMFDUJPOPGTFSWJDFTUPCFUFTUFECBTFEPO EFQFOEFODZNBQBOEUFTUFYFDVUJPO  %FUFDUJPOPGVOFYQFDUFEGBJMVSFTUISPVHI QFSJPEJDUFTUFYFDVUJPO

  24. LTͷ࢓૊ΈΛ࢖ͬͨ ࣮૷ͷΞΠσΞ

  25. 0WFSWJFX Github (on-premise) App ArgoCD Kubewatch Tester Notifier Dependency Mapper

    Slack/ Alertmanager
  26. LT "SHP$% ,VCFXBUDI "QQMZUIFNBOJGFTUPOUIF(JUSFQP .BOBHFNFOU/PUJGJDBUJPO  3VOXJUIUIFNFDIBOJTNUPFYFDVUFEFQMPZNFOU CBTFEPOBQPTUPO(JUIVC3FMFBTF  0QFSBUFBXFCIPPLCBTFEOPUJGJDBUJPOTFSWJDFUP

    SFDFJWFBOEQSPQFSMZOPUJGZUIFSFTVMUTPG SFTPVSDFNPOJUPSJOHCZ,VCFXBUDIBOEUFTUJOH
  27. LT "SHP$% ,VCFXBUDI "QQMZUIFNBOJGFTUPOUIF(JUSFQP .BOBHFNFOU/PUJGJDBUJPO  &OGPSDFNFUBEBUBTFUUJOHTXJUI"ENJTTJPO $POUSPM

  28. 5FTUTZTUFNCBTFEPO3FDPODJMFMPPQ 0QFSBUPSNBOBHFUFTUFYFDBOESFTVMUTGSPN SFTPVSDFDIBOHFBOEQFSJPEJDFWFOUT  .BJOUBJOBMJTUPGKPCNBOJGFTUTGPSUFTUJOHBTB DVTUPNSFTPVSDFBTTPDJBUFEXJUIUIFTFSWJDF  $POUSPMFYFDVUJPOPGUFTUTGPSSFMBUFETFSWJDFT CBTFEPOUIFEFQFOEFODZNBQXIFOBOPUIFS TFSWJDFEFQMPZNFOUFWFOUPDDVST

  29. .FUSJDTCBTFEOPUJGJDBUJPOTZTUFN .POJUPSTQSFEFUFSNJOFELVCFTUBUFNFUSJDT JOVOJUTPG%FQMPZNFOUBOEOPUJGJFT CBTFEPOTVEEFODIBOHFT  *TTVFBMFSUGSPN"MFSUNBOBHFSUPUIF BGPSFNFOUJPOFEXFCIPPLCBTFEOPUJGJDBUJPOTZTUFN  0GDPVSTF OPUJGJDBUJPOXJMMCFFYFDVUFECBTFEPO

    UIFEFQFOEFODZNBQ
  30. 0WFSWJFX Github (on-premise) App ArgoCD Kubewatch Tester Notifier Dependency Mapper

    Slack/ Alertmanager
  31. 7FSEBͰͷ ཁૉٕज़ͷ ಋೖࣄྫ

  32. 0QFSBUJPOXJUI3FDPODJMF-PPQ .BOBHFUIFTUBUFPG/'7JOTUBODF BTLTSFTPVSDF • 3PVUFS 71/(BUFXBZ FUDDBOCFPQFSBUFEBT QVSFLTSFTPVSDFT • &GGPSUTUPEFGJOFLTBTUIFGPVOEBUJPOGPS

    FYFDVUJOHTUSVDUVSFEEBUBTUPSFTBOEBDUJPOT SFMBUFEUPUIBUEBUB
  33. None
  34. 0QFSBUJPOXJUI3FDPODJMF-PPQ .BOBHFUIFTUBUFPG/'7JOTUBODF BTLTSFTPVSDF https://www.janog.gr.jp/meeting/janog48/wp- content/uploads/2021/07/janog48-LINENFV- %E5%9F%8E%E5%80%89%E5%BC%98%E6%A8%B9.pdf

  35. %QMBOFEFQMPZNFOULTMJ[BUJPO &OBCMJOHEBUBQMBOFTUIBUNVTUCFEFQMPZFE EJSFDUMZUPCBSFNFUBMUPCFEFQMPZFEGSPN LT • .BJOUBSHFU)ZQFSWJTPSTPG0QFO4UBDL • $VSSFOUMZ UIFGPDVTJTPOEFDMBSBUJWF EFQMPZNFOUBOEQSPQFSGPSLJOHPG"OTJCMF

    FYFDVUJPO
  36. %QMBOFEFQMPZNFOULTMJ[BUJPO &OBCMJOHEBUBQMBOFTUIBUNVTUCFEFQMPZFE EJSFDUMZUPCBSFNFUBMUPCFEFQMPZFEGSPN LT https://engineering.linecorp.com/ja/blog/internship- report20-hypervisor-update-system-with-kubernetes/

  37. 6TF"SHP$% 6TJOH"SHP$%GPS,BB4"1*EFQMPZNFOU • &GGJDJFOUQSFQBSBUJPOVTJOHBXFBMUIPGSFMBUFE LTSFMBUFEUPPMT • *ODSFBTJOHUIFWJTJCJMJUZBOETQFFEPGQSPHSFTT CZNPWJOHBXBZGSPNBQSPDFEVSBMBQQSPBDI

  38. ·ͱΊ

  39. %FQMPZNFOUCZEFDMBSBUJPO • %FQMPZNFOUUBSHFUTDBOCFBCTUSBDUFEBOE NBOBHFEBTLTSFTPVSDFTUPFOGPSDFUIF BTTJHONFOUPGNFUBEBUB SFTPVSDFMJNJUT FUD • 3FQSFTFOUJOHEFQMPZNFOUBTBDIBJOPG QSPDFTTFTXJUITNBMMSFTQPOTJCJMJUJFTCBTFEPO

    SFTPVSDFT SBUIFSUIBOBTFSJFTPGQSPDFEVSFT  NBLFTJUFBTJFSUPNBOBHFSPMMCBDLBOEPUIFS TDFOBSJPT
  40. 1PTUEFQMPZNFOU UFTUT CZEFQFOEFODZNBQ • *OBEEJUJPOUPUIFIFBMUIPGUIFTFSWJDF JUBMTP OFFETUPCFDIFDLFEJGTPNFSPVUJOFBENJO PQFSBUJPOTBSFQPTTJCMF • #BTFEPOUIFEFQFOEFODZNBQ

    SVOFWFOUCBTFE UFTUTPOPUIFSTFSWJDFTUIBUEFQFOEPOUIF TFSWJDFUPCFEFQMPZFE
  41. %FQMPZNFOUOPUJGJDBUJPO CZEFQFOEFODZNBQ • /PUJGZSFMBUFETFSWJDFTPGEFQMPZNFOUGBJMVSFT  UFTUGBJMVSFT FUDVTJOHEFQFOEFODZNBQTUP TUSFBNMJOFJNQBDUDIFDLT • *UDBOCFBHFOFSJDGSBNFXPSLGPSSPVUJOFIFBMUI

    DIFDLT FTQFDJBMMZGPSFWFOUCBTFEBOEQFSJPEJD OPUJGJDBUJPOPGUFTUSFTVMUT
  42. 5IFQSFNJTFJTʜ • :PVTIPVMEDIFDLUIFJNQMFNFOUBUJPOPGUIF TFSWJDF)PXTIPVMEJUEFQFOEPOPUIFS TFSWJDFT • 8IBULJOEPGDIFDLTTIPVMECFQFSGPSNFEJO TUBHJOHBOEVOJUUFTUTUPQSFWFOUQSPCMFNTJO QSPEVDUJPOEFQMPZNFOUT 1FSGFDUEFQMPZNFOUJTOPUBTJMWFSCVMMFU

  43. THANK YOU