pixiv TECH SALON ( https://techsalon.pixiv.co.jp/ )の発表資料です。
εϖοΫΛ্͛ͯΫϥυͰԥΔCIpixiv.incsue445
View Slide
• HN: sue445• ށ੶ωʔϜ: Go Sueyoshi• 20187݄ɹϐΫγϒೖࣾ‣ ࠓճͷొஃऀͷதͰൺֱత৽ࢀऀ• Πϯϑϥ෦• ϑϧελοΫΩϡΞΤϯδχΞ‣ ϓϦΩϡΞͷΧόϨοδ100%ܧଓதgo versionʢࣗݾհʣ
• SUE ≒ SRE• SREʹࣅͯΔࣄΛͬͯΔgo versionʢࣗݾհʣ
https://rubykaigi.org/2019/speakersʲએʳRubyKaigi 2019ొஃ༧ఆ
ͷϐΫγϒͰͷϛογϣϯ
ʮશͯͷख࡞ۀΛࣗಈԽ͢ΔʯͷϐΫγϒͰͷϛογϣϯ
• ػցʹͤΒΕΔ͜ͱػցʹશ෦ͤͯɺਓ͔ؒ͠Ͱ͖ͳ͍͜ͱΛਓ͕ؒΔ͖• ݱঢ়·ͩ΄Ͳԕ͍ͷͰ·ͣCIͰձࣾΛϋοΫ͍ͯ͠Δ• ʮಇ͔ͳ͍ͨΊʹશྗͰಇ͍͍ͯΔʯʢʁʣʮશͯͷख࡞ۀΛࣗಈԽ͢Δʯ
• ϐΫγϒͷCIࣄͱݱঢ়ͷCIͷ• Ͳ͏ͬͯվળ͔ͨ͠• ϐΫγϒͳΒͰͷۤ࿑ࠓ͢͜ͱ
• CIϨϕϧʢεϖοΫʣΛ্͛ͯཧʢΫϥυʣͰԥΔͷ͕େਖ਼ٛ• SaaS͍͍͚ͲࣗલͰΔͷӡ༻ܦݧ͕ஷ·ΔͷͰ͓͢͢Ί‣ ͨͩ͠ສਓʹ͓͢͢Ί͠ͳ͍࠷ॳʹ·ͱΊ
• େલఏ‣ ྺ࢙తܦҢʹΑΓGitHub.comͱGitLabʢΦϯϓϨʣ͕྆ํΘΕ͍ͯΔϐΫγϒͷCIࣄ
• GitHub.com‣ RailsܥGitHubΛ͍ͬͯΔ‣ ྫʣBOOTH, pixivFACTORY, pixiv PAY, pixivίϛοΫ‣ CI: ͍͍ͩͨશ෦CircleCIΛͬͯΔϐΫγϒͷCIࣄ
• GitLabʢΦϯϓϨʣ‣ pixivຊମʢ͍ΘΏΔΈΜͳ͕Αͬͯ͘Δpixivʣɺpixivຊମͱີ݁߹͍ͯ͠ΔपลαʔϏεɺVRoid Studio‣ CI: GitLab CI, Jenkins‣ ࠓGitLab CIͷ͕ϝΠϯͰ͢ϐΫγϒͷCIࣄ
• ͦͷଞʢϦϙδτϦඇґଘʣ‣ iOS/AndroidΞϓϦͷCIʹBitriseΛར༻‣ ৄ͍͜͠ͱɿϞόΠϧΞϓϦͷCIΛBitriseʹͯ͠1͕ܦͪ·ͨ͠ - pixiv inside- https://inside.pixiv.blog/kwzr/6190ɹϐΫγϒͷCIࣄ
• VRoid StudioʢUnityʣͰϏϧυͷ͕͔͔͍࣌ؒͬͯͨ‣ Ϗϧυ1ճ100ʢʂʣ‣ ϋΠεϖοΫͳGitLab Runner͕΄͍͠ͱ͍͏ཁ‣ ͔ͤͬ͘ͳͷͰVRoid Studioઐ༻Ͱͳ͘ɺࣾGitLabͷશମͰ͑ΔRunnerʹͨ͠‣ CIͷվળ݁ՌɺΈΜͳͷੜ࢈ੑόΫ্͕Γʹͳͬͯͤʹͳͬͯ·͢CIͷ
• Before‣ MacMini‣ ྻͰ100ܶతϏϑΥʔΞϑλʔ
• Before‣ MacMini‣ ྻͰ100• After‣ EC2 c5.2xlargeΠϯελϯεʢvCPU 8, ϝϞϦ16GiBʣ‣ 3ฒྻͰ25ܶతϏϑΥʔΞϑλʔ
• લͰ5ɺޙͰ֤20લޙ• 100 -> 25ͳͷͰ4ഒͷੜ࢈ੑܶతϏϑΥʔΞϑλʔ
Ͱ͓ߴ͍ΜͰ͠ΐ͏ʁ
AWSඅ༻݄300υϧ͘Β͍Ͱ͓ߴ͍ΜͰ͠ΐ͏ʁ
ࢀߟɿ1݄ͷඅ༻
• https://docs.gitlab.com/runner/configuration/runner_autoscale_aws/‣ AWSͷεϙοτΠϯελϯεͱdocker machineΛར༻ͨ͠ΦʔτεέʔϧRunnerΛߏங͢ΔͨΊͷެࣜυΩϡϝϯτ• https://www.m3tech.blog/entry/advent-calendar-2018-2‣ M3͞ΜͷςοΫϒϩάɻຊޠͷઆ໌͕ΉͬͪΌৄ͍͠‣ ʮGitLab Runner AWS spot instanceʯͰάάΕ͍͘ΒͰࢿྉग़ͯ͘Δৄ͍͜͠ͱ
• άάΕ͔Δ͜ͱʹ͍ͭͯͯ͠͠ΐ͏͕ͳ͍ͷͰɺϐΫγϒͳΒͰͷɺۤ࿑ɺӡ༻ݟͳͲΛόόʔϯͱհ͠·͢ʂϐΫγϒGitLabͱAWS Runner
• ͜Εݟ͔ͯΒͳ͍ͱࢥ͏ͷͰ͔͍ͭ·ΜͰઆ໌AWSͷGitLab Runnerͷશମ૾
• εϙοτΠϯελϯε• Docker Machine• Ansible• Terraform• Packer• Serverless Frameworkओͳొਓ
• ҆͑͘ΔEC2ͷ༨Πϯελϯε‣ ੑೳΦϯσϚϯυΠϯελϯεʢεϙοτΠϯελϯε͡Όͳ͍ී௨ͷͭʣͱશ͘ಉ͡ͰɺՁ֨࠷େ9ׂҾ‣ ྫʣc5.2xlargeͩͱΦϯσϚϯυ͕0.428USD/࣌ؒɺεϙοτ͕0.0779USD/࣌ؒલޙ• εϙοτΠϯελϯεͷՁ֨มಈ͢ΔͷͰɺࣄલʹઃఆͨ͠ೖࡳՁ֨Λ͑Δͱ࡞Εͳ͍‣ ଟগՁ্͕͕֨ͬͯೖࡳΤϥʔʹͳΒͳ͍Α͏ͳֹۚͰೖࡳ͢Δͷ͕͓͢͢ΊεϙοτΠϯελϯε
• ΞΧϯτ͝ͱʹ࡞Ͱ͖ΔεϙοτΠϯελϯεͷ্ݶ͕ܾ·ͬͯΔʢσϑΥϧτ20ʣ• εϙοτΠϯελϯεΛআͨ͠Β·ͨ࡞Ͱ͖Δͣͳͷ͕ͩɺͨ·ʹআࡁͷΠϯελϯε͕࡞্ݶΛѹഭ͢Δ͜ͱ͕͋Δ‣ Α͘ʮεϙοτΠϯελϯεͷΰϛʯͱݴͬͯΔ• ϝΠϯͰ͏ΠϯελϯελΠϓҰʹཱͯΔΠϯελϯεΑΓଟΊʹ্ݶ؇ਃΛͨ͠ํ͕͍͍ɻʢϐΫγϒͩͱ10ฒྻʹରͯ͠100ݸͰ্ݶ؇ʣεϙοτΠϯελϯεʢҙʣ
• ϦϞʔταʔό্ʹdockerίϯςφΛߏங͢ΔΈ• DockerίϚϯυͷ࣮ߦݩͱίϯςφͷ࡞ઌ͕ผ• http://docs.docker.jp/machine/overview.htmlDocker Machine
• ௨ৗDockerͱ͍͏ͱͬͪ͜Λࢦ͢߹͕ଟ͍ʢDockerίϚϯυͷ࣮ߦݩͱίϯςφͷ࡞ઌ͕ಉ͡ʣ‣ http://docs.docker.jp/machine/overview.htmlൺֱ༻ɿDocker Engine
GitLab Runner + Docker MachineGitLabRunnerDockerMachineΦϯϓϨڥec2ec2ec2
• ͜ͷߏͷϝϦοτ‣ Docker EngineʢαʔόͷϩʔΧϧʹίϯςφΛཱͯΔʣͱҧͬͯɺॏ͍δϣϒ͕1ͭ͋ͬͨ࣌ʹଞͷδϣϒʹӨڹ͕ແ͍‣ ϋΠεϖοΫͳϚγϯΛδϣϒ1ͭͰઐ༗Ͱ͖Δ‣ Runner͕Α͠ͳʹΦʔτεέʔϧͯ͘͠ΕΔͷͰɺδϣϒ͕ੵ·Εͳ͚ΕEC2উखʹফ͍͑ͯ͘GitLab Runner + Docker Machine
• αʔόͷϓϩϏδϣχϯάπʔϧ• طଘͷRunnerʢDocker Engineར༻ʣͷplaybook͕͋ͬͨͷͰɺ৽͘͠ΦʔτεέʔϧRunner༻ͷઃఆΛ࡞ͬͨAnsible
• AWSͳͲͷΫϥυͷߏཧπʔϧ• VPC, S3, IAMͳͲͷϓϩϏδϣχϯάͰར༻• GitLabͷMergeRequestͰterraform planʢdry runʣΛ͠ɺmasterʹϚʔδ͞ΕͨΒapplyʢຊ࣮ߦʣ͢Δͱ͍͏ΠϯϑϥCIͰར༻͍ͯ͠ΔTerraform
• αʔόͷΠϝʔδʢAWSͳΒAMIʣΛ࡞ΔͨΊͷπʔϧ• ༧ΊϐΫγϒGitLab༻ͷઃఆΛೖΕͯAMIΛ࡞͠ʢޙड़ʣɺͦΕΛRunnerͰ͍ͬͯΔPacker
• ϩʔΧϧ։ൃ༻ʹVagrantಋೖ• VagrantPackerͰͷϓϩϏδϣχϯάʹmitamaeΛ͍ɺϓϩϏδϣχϯάͨ͠༰ΛServerspecͰςετΛ͍ͯ͠Δ• ͪΖΜGitLab CIͰCI/CD͍ͯ͠Δ• ৄ͍͜͠ͱશ෦ https://sue445.booth.pm/items/1033989 ʹॻ͍ͯΔʢεςϚʣ‣ CircleCIΛGitLab CIʹม͑ͨҎ֎αϯϓϧϦϙδτϦΛ΄΅ؙύΫϦͰ͖ͯΉͬͪΌศརͩͬͨPacker + Vagrant + mitamae + Serverspec
• ʢAWS͚ͩʹݶͬͯݴ͑ʣLambdaͱͦͷपลϦιʔεΛ͍͍ײ͡ʹཧͯ͠σϓϩΠ͢ΔͨΊͷπʔϧ• TerraformͰLambdaΛཧ͢Δͷ৭ʑͱେมͳͷͰɺͦ͜Λϥοϓͯ͘͠Ε͍ͯΔͷ͕خ͍͠• ࠓճɺAWSͷϞχλϦϯάΛLambda + RubyͰ࡞ͬͯɺServerless FrameworkͰཧͯ͠ΔServerless Framework
• ىಈͯ͠ΔΠϯελϯεͱGitLabCIͷδϣϒΛLambdaͰूܭͯ͠CloudWatchͰ͍͍ײ͡ʹՄࢹԽ͍ͯ͠Δ• ͦͷ͏ͪιʔεΛެ։͍͕ͨ͠RubyKaigiͷ४උ͕͕͕͕͕Lambda + Ruby + Serverless FrameworkͰ࡞ͬͨࢹπʔϧ
• AWS͔ΒGitLabʹΞΫηεͰ͖ΔΑ͏ʹͨ͠• CIͷδϣϒؒͰେ༰ྔϑΝΠϧΛड͚ͤΔΑ͏ʹͨ͠• Runnerͷىಈ࣌ؒνϡʔχϯά• εϙοτΠϯελϯεރׇରࡦϐΫγϒͳΒͰͷͳͲ
• લఏ‣ ϐΫγϒGitLabssh༻ͷURLࣾDNSͰ͔͠ղܾͰ͖ͳ͍ͷͰɺAWS͔ΒͩͱΞΫηεͰ͖ͳ͍- Ծʹ໊લղܾͰ͖ͯࣾ֎͔ΒͷΞΫηεΛड͚ΔGatewayαʔόʹผͷೝূ͕ڬ·ͬͯΔͷͰΞΫηεͰ͖ͳ͍‣ httpͷURLࣾ֎͔ΒͩͱGoogleͷOAuthೝূ͕ڬ·ΔͷͰAWS͔ΒGitLabͷAPI͕ୟ͚ͳ͍AWS͔ΒGitLabʹଓͰ͖ΔΑ͏ʹͨ͠
• RunnerͰ͏AMIʹ༧Ί /etc/hosts Λম͍͓͖ͯɺࣾDNSͷ໊લղܾ͕Ͱ͖ΔΑ͏ʹͨ͠• ͜͜ͰPackerΛར༻ղܾࡦ1: ໊લղܾࡁͷ /etc/hostsΛম͘
• GatewayαʔόͷiptablesͰAWS͔Βͷ௨৴ΛڐՄ͢ΔΑ͏ʹͨ͠• ͦͷ··ΔͱEC2Πϯελϯεͷ࡞Γ͠ͷʹૹ৴ݩͷIPΞυϨε͕มΘͬͯGatewayαʔόͰڐՄ͠Α͏͕ͳ͍ͷͰɺNAT GatewayΛͬͯAWSͷVPC͔ΒGatewayαʔόग़Δ࣌ʹૹ৴ݩͷIPΞυϨεΛݻఆԽͨ͠• NAT Gatewayʹ༩ͨ͠Elastic IPΛGatewayαʔόͰڐՄ͍ͯ͠Δղܾࡦ2: ΦϯϓϨͷຯํNAT Gateway
• ಛఆͷIPΞυϨε(ࠓճͷ߹Gateway)ͷ௨৴Λશ෦NAT GatewayΛ௨͢͜ͱʹΑΓɺૹ৴ݩͷIPΞυϨεΛݻఆԽ͍ͯ͠Δ"[NAT GatewayͷΠϝʔδSubnetGitLabΦϯϓϨڥAmazonVPCec2
• ಛఆͷIPΞυϨε(ࠓճͷ߹Gateway)ͷ௨৴Λશ෦NAT GatewayΛ௨͢͜ͱʹΑΓɺૹ৴ݩͷIPΞυϨεΛݻఆԽ͍ͯ͠Δ"[NAT GatewayͷΠϝʔδSubnet SubnetNAT GatewayGitLabΦϯϓϨڥGatewayec2*1ΞυϨεΛݻఆAmazonVPC
• NAT GatewayͷElastic IPΛGatewayͷnginxͰڐՄͯ͠Δ͜ͱʹΑΓɺGoogleͷOAuthೝূΛճආ͢ΔΑ͏ʹͨ͠ղܾࡦ3: httpܦ༝ͷΞΫηεNAT GatewayΛ௨͢
CIͷδϣϒؒͰେ༰ྔϑΝΠϧΛड͚ͤΔΑ͏ʹͨ͠• ܦҢ‣ VRoid StudioνʔϜͷਓʮδϣϒ͕ऴΘͬͯGitLabʹArtifactsΛΞοϓϩʔυ͠Α͏ͱ͢ΔͱΤϥʔʹͳΔʯ‣ sue445ʮGitLabͷϩάݟͨΒ413Τϥʔ(Request Entity Too Large)ग़ͯ·͢ͶɻnginxͷΞοϓϩʔυ੍ݶʹҾ͔͔ͬͬͯΔΑ͏ͳͷͰGitLabଆͷΞοϓϩʔυαΠζ؇Ί·͕͢zipʹݻΊͨ࣌ͷϑΝΠϧαΠζͲΕ͘Β͍͋Γ·͢ʁʯ
CIͷδϣϒؒͰେ༰ྔϑΝΠϧΛड͚ͤΔΑ͏ʹͨ͠VRoid StudioνʔϜͷਓʮ4GB΄͍͠Ͱ͢ʯʢݪจϚϚʣsue445ʮʯ
• ͖ͬ͞ͷGatewayαʔόࣾ։ൃશൠͰ͍ͬͯΔͷͰɺGatewayͷଳҬ͕٧·Δͱࣾͷ։ൃ͕શ෦ࢮ͵• RailsΞϓϦͰ4GBͷϑΝΠϧͷΞοϓϩʔυͱ͔͢Δͱunicorn͕ṫࢮʹͳΔະདྷ͔͠ݟ͑ͳ͍‣ ʣGitLabRailsΞϓϦ• AWS͔Β֎ʢΦϯϓϨͷGitLabʣʹग़͍ͯ࣌͘ͷ௨৴අ͕͔͔Δཁ͢Δͱී௨ͷWebΞϓϦʹڊେϑΝΠϧΛup͢Δͷਏ͍
• ΞϓϦ͔ΒಛผͳઃఆΛͤͣʹS3ʹΞΫηεͰ͖ΔΑ͏ʹͨ͠• Γํ‣ ಛఆͷS3όέοτͷΈʹΞΫηεͰ͖ΔIAMϩʔϧΛ࡞‣ Docker Machine͕EC2ΠϯελϯεΛىಈ͢Δ࣌ͷΠϯελϯεϓϩϑΝΠϧʹ͜ͷIAMϩʔϧΛઃఆ‣ ͜͏͢Δ͜ͱͰRunnerଆawscli͚ͩ༻ҙ͍ͯ͠ΕόέοτʹಡΈॻ͖Ͱ͖ΔɻʢΞϓϦଆͰΞΫηεΩʔͳͲͷઃఆෆཁʣ• ಉҰϦʔδϣϯͩͱEC2ͱS3ͰϑΝΠϧΓऔΓ͢Δ࣌ͷ௨৴අ͕͔͔Βͳ͍ͷخ͍͠ղܾࡦ: S3ʹΞοϓϩʔυ͢ΔʢԦಓʣ
• લఏ‣ GitLab RunnerʹOffPeakͱ͍͏ઃఆ͕͋ͬͯɺʮۀ࣌ؒৗʹEC2ΠϯελϯεΛ1ػ͢Δ͕ɺۀ࣌ؒ֎ػͤ͞ͳ͍͜ͱͰίετΛԼ͛Δʯͱ͍͏͜ͱ͕Մೳ‣ https://docs.gitlab.com/runner/configuration/autoscale.html‣ https://docs.gitlab.com/runner/configuration/advanced-configuration.htmlRunnerͷىಈ࣌ؒνϡʔχϯά
• લఏ‣ ϐΫγϒGitLabͷ߹ԼهͷΑ͏ͳઃఆ‣ ฏ10:00ʙ18:59εϙοτΠϯελϯεΛ࠷1ػͯ͠ɺδϣϒ͕ੵ·ΕͨΒଈϏϧυ։࢝Ͱ͖ΔΑ͏ʹͯ͠Δ‣ ۀ࣌ؒ֎ɾਖ਼݄ɾGWεϙοτΠϯελϯεΛػͤͣ͞ɺδϣϒ͕ੵ·Εͨ࣌ͰΠϯελϯεΛىಈRunnerͷىಈ࣌ؒνϡʔχϯά
• ‣ δϣϒ͕ੵ·Ε͔ͯΒ࣮ࡍʹϏϧυ͕࢝·Δ·Ͱ࠷େ4͘Β͍͔͔ͬͯͨͷͰؤுͬͯνϡʔχϯά͔ͨͬͨ͠‣ ۀ࣌ؒͩͱEC2͕1Ҏ্ػͯ͠ΔͷͰؾʹͳΒͳ͍͚Ͳఆ࣌ա͗ͯpushͨ࣌͠ʹRunnerͷ࣮ߦ·Ͱ͕͔͔࣌ؒΔ‣ ىಈ͕͍ΑΓ͍ํ͕͍͍‣ ͞ਖ਼ٛɺ࣌ۚͳΓRunnerͷىಈ࣌ؒνϡʔχϯά
• Docker MachineϦϞʔτʹdocker͕Πϯετʔϧ͞Ε͍ͯͳ͍࣌ͷΈdockerΠϯετʔϧͯͨ͠ͷͰɺdocker͕ΠϯετʔϧࡁͳΒ͜ͷॲཧΛεΩοϓͰ͖Δͱ౿Μͩ‣ https://github.com/docker/machine/blob/v0.16.0/libmachine/provision/utils.go#L30‣ Docker MachineͰىಈ͢ΔAMIʹdockerΛΠϯετʔϧ͓͚ͯͩ͘͠Ͱ4͔Β4ʹॖͰ͖ͨղܾࡦ: AMIʹ༧ΊdockerΛೖΕ͓ͯ͘
• ݸਓతʹ͏ͪΐͬͱॖ͔͕ͨͬͨ͠ɺ͜ΕҎ্Ζ͏ͱ͢ΔͱDocker MachineຊମΛ͍͡Δඞཁ͕͋ͬͨͷͰஅ೦• Machine࡞࣌ͷϩάΛશ෦ಡΜ͚ͩͲͲΕ͔1ͭͷॲཧ͕͍ͱ͍͏ΑΓɺࡉ͔͍ॲཧ͕ͨ͘͞Μ͋ͬͯνϦπϞͰ4͔͔ͬͯΔͱ͍͏ҹͩͬͨղܾࡦ: AMIʹ༧ΊdockerΛೖΕ͓ͯ͘
• લఏ‣ εϙοτΠϯελϯεAWS্ͷ༨ΠϯελϯεΛ҆͑͘ΔΈͳͷͰɺAWSશମͷεϙοτΠϯελϯεͷࡏݿ͕ͳ͚ΕىಈͰ͖ͳ͍‣ ͔͠͠εϙοτΠϯελϯε͕ރׇͨ࣌͠ʹࣾͷ։ൃ͕શʹࢭ·Δͷ͖͍ͭεϙοτΠϯελϯεރׇରࡦ
• εϙοτΠϯελϯε͕ރׇͨ࣌͠ʹผͷΠϯελϯελΠϓAZͰࢼ͢ͳͲͷεϙοτΠϯελϯεΨνϟΛΔΑΓɺͬ͞͞ͱΦϯσϚϯυʹશৼΓͨ͠ํ͕ૣ͘෮چͰ͖Δͱ͍͏அ• ࠓ·Ͱͷܦݧ্ɺͲ͔ͬ1ϲॴͰεϙοτΠϯελϯε͕ރׇͯ͠Δͱͦͷपล͍͍ͩͨރׇͯ͠Δҹղܾࡦ: ΦϯσϚϯυΠϯελϯεΛ͏RunnerΛ࡞ͬͨ
• ΦϯσϚϯυΠϯελϯεΛ͏RunnerϗοτελϯόΠʢRunnerͱͯ͠ಈ͍ͯΔ͕ཧը໘্ͰແޮͳͷͰδϣϒׂ͕ΓৼΒΕͳ͍ʣͰ༻ҙ͓͍ͯͯ͠ɺGitLabͷཧը໘্Ͱ༗ޮԽͨ࣌͠ͷΈΘΕΔΑ͏ʹͯ͠Δղܾࡦ: ΦϯσϚϯυΠϯελϯεΛ͏RunnerΛ࡞ͬͨ
• εϙοτΠϯελϯεΛ׆༻͢Δ͜ͱͰϋΠεϖοΫͳCIڥΛՁ֨Ͱӡ༻Մೳ• CIʹݶΒͣ͜Ε͔ΒશྗͰࣗಈԽ͍͖ͬͯ·͢·ͱΊ
sue445
shibuiwilliam
naohachi89
keiko713
takesection
niftycorp
yoshiitaka
a_bicky
onecareer_tech
k1low
nayuts
dahatake
philhawksworth
shpigford
skipperchong
lara
philnash
morganepeng
bkeepers
marcelosomers
trishagee
addyosmani
pedronauck