トレタのインフラ運用

by Hiroaki Sano

Slide 1

Slide 1 text

τϨλͷΠϯϑϥӡ༻ ʲτϨλºϓϨΠυʳ5FDI#MPH%FFQ%JWF.FFUVQ )JSPBLJ4BOP

Slide 2

Slide 2 text

NF • Name: • ࠤ໺༟ষ(Hiroaki Sano) • Personal website: • https://hiroakis.com/blog/ • Company: • NEC Soft, Ltd. (2006/4-) • CyberAgent, Inc. (2011/3-) • Toreta, Inc. (2014/11-)

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

No content

Slide 5

Slide 5 text

τϨλ • ҿ৯ళ޲͚༧໿؅ཧΞϓϦέʔγϣϯΛiPadΞϓϦͱͯ͠ఏڙ • ҰൠϢʔβ޲͚ʹ΢Σϒ༧໿΋ఏڙ • ސ٬ɿϨετϥϯɺډञ԰ͳͲͷҿ৯ళ • ྫ͑͹… • Զͷגࣜձ༷ࣾ(ԶͷΠλϦΞϯɺԶͷϑϨϯν…etc) • גࣜձࣾେঙ༷(ঙ΍ɺ΍Δ͖஡԰…etc) • ΤʔɾϐʔΧϯύχʔ༷(௩ా೶৔…etc) • ϩʔετϗʔε༷ • etc • ϩʔϯν͸2013೥12݄ • ಋೖళฮ਺ɿ6000ళฮʙ • ళฮ͋ͨΓͷֹ݄՝ۚ • ίʔϙϨʔτΧϥʔ͸྘ • ΦϑΟε͸ौ୩ -> ໨ࠇ -> ܙൺण -> ޒ൓ా

Slide 6

Slide 6 text

Slide 7

Slide 7 text

Slide 8

Slide 8 text

τϨλͷίϯηϓτ • ҿ৯ళͷ༧໿؅ཧɺސ٬؅ཧΛITԽ • ҿ৯ళͰಇ͘ਓͷۀ຿ͷࣗಈԽ • ༧໿ࣄނΛ๷͙

Slide 9

Slide 9 text

ࠓ೔͸ٕज़ϒϩάਂ۷Γձ ͱ͍͏͜ͱͰʜ

Slide 10

Slide 10 text

τϨλ։ൃऀϒϩά IUUQUFDIUPSFUBJO

Slide 11

Slide 11 text

No content

Slide 12

Slide 12 text

ࢲͷॻ͍ͨهࣄ • Raspberry PiͱԹ౓ɾ࣪౓ɾޫηϯαʔͰΦϑΟεͷঢ়گΛՄࢹԽ͢Δ • http://tech.toreta.in/entry/2016/02/15/153203 • τϨλͷΠϯϑϥӡ༻ɺࢧ͍͑ͯΔಓ۩(Packer, Terraform, Serverspec, Ansible, Roadworker, Circle CI)ɺߟ͑ํ • http://tech.toreta.in/entry/2016/04/14/143248 • ϝʔϧͷ഑৴ঢ়گΛՄࢹԽɺ௥੻͢Δ • http://tech.toreta.in/entry/2016/04/20/153852 • Engineyard͔ΒAWSʹҠઃͯ͠Auroraͷӡ༻Λ։࢝ͨ͠ • http://tech.toreta.in/entry/2016/06/16/114919

Slide 13

Slide 13 text

ࠓ೔࿩͢͜ͱ • ओʹAWSͰͷΠϯϑϥӡ༻ʹ͍ͭͯ • τϨλͷAPIαʔόͷΤίγεςϜ • ओʹϒϩάωλʹొ৔ͨ͠ࣄฑΛத৺ʹ… • Terraform • Roadworker • Packer • Ansible • Serverspec • Aurora • Circle CI • ࠓޙʹ͍ͭͯ • ٕज़తʹਅ৽͍͠΋ͷ͸ͳ͍Ͱ͕͢ࢀߟʹ͍͚ͯͨͩ͠Ε͹…

Slide 14

Slide 14 text

τϨλ"1*ͱͦͷपลͷΤίγεςϜ

Slide 15

Slide 15 text

Amazon Route 53 Amazon RDS(Aurora) api  worker Redis memcached Amazon SNS Amazon SQS τϨλ"1*ͱͦͷपลͷΤίγεςϜ FAX/SMS… Amazon S3 ը૾/Ի੠… ଞαϒγεςϜ΁ͷ௨஌ ϩά ϝʔϧ fax, sms API worker fluentd

Slide 16

Slide 16 text

τϨλ"1*ͱͦͷपลͷΤίγεςϜ • Amazon Web Services • 5݄ʹEngineyard͔ΒҠઃ(ϒϩάهࣄͷ௨Γ) • Appαʔό: APIɺWorkerಉډ. nginx, Rails(unicorn), sidekiqͰ࣮૷. • Redis: WorkerͷΩϡʔΠϯά༻ • memcached: Ωϟογϡ • RDS Aurora: ϝΠϯͷσʔλϕʔε • Google Cloud Platform • BigQuery: fluentd͔Βϩάͷసૹ • ιʔϦʔαʔό΍Ұ෦ͷόοναʔό: GCEͷݕূΛ݉Ͷͯͪ͜Βʹߏஙͯ͋͠Δ • Monitoring • Mackerel: Ϧιʔε؂ࢹɺϝτϦΫε • Pingdom: ΤϯυϙΠϯτ؂ࢹ • Pagerduty: ΞϥʔτରԠͷεέδϡʔϦϯάɺి࿩௨஌ • CI • Circle CI: CI͓ΑͼσϓϩΠɺ͓ΑͼΦϖϨʔγϣϯͷத৺

Slide 17

Slide 17 text

app  worker Redis Aurora Writer memcached app  worker Redis memcached Availability Zone A 71$ઃܭ public subnet Availability Zone C Aurora Reader public subnet 10.0.0.0/16 10.0.0.0/24 10.0.1.0/24

Slide 18

Slide 18 text

71$ઃܭ • ֤ίϯϙʔωϯτ͸ϚϧνAZͰ഑ஔ • 10.0.X.0/24 -> AZ-A • 10.0.Y.0/24 -> AZ-C • ύϒϦοΫαϒωοτͷΈ • RDSͳͲ΋ύϒϦοΫαϒωοτʹஔ͘ • ݎ࿚ੑ͸ηΩϡϦςΟάϧʔϓͰ୲อ • ࡶ͡Όͳ͍͔ʁͱͨ·ʹݴΘΕΔ • ݱঢ়ͷ࡞ΓͰผʹࠔ͍ͬͯͳ͍ • ࠔͬͨΒ࡞Γͳ͓͢

Slide 19

Slide 19 text

5FSSBGPSN

Slide 20

Slide 20 text

5FSSBGPSN • https://www.terraform.io/ • Hashicorp੡ϓϩμΫτ • AWS΍GCPͳͲ֤छΫϥ΢υͷঢ়ଶΛίʔυͰهड़͢Δ΋ͷ • HCLͱ͍͏jsonϥΠΫͳϑΥʔϚοτͰهड़ ग़య: https://www.terraform.io/

Slide 21

Slide 21 text

5FSSBGPSN • VPC, SecurityGroup, EC2Λ؅ཧର৅ͱ͍ͯ͠Δ • EC2ߏங࣌͸Environment, Role, ServiceλάΛ෇༩͢Δ • ͜ΕΒ͸σϓϩΠ΍ϓϩϏδϣχϯάͰ࢖͏ • AWSͷ࢖͍ํͱͯ͠యܕతͱ͍͏͔ݹయతͳखஈ

Slide 22

Slide 22 text

ͳͥ5FSSBGPSN͔ʁ • CloudFormationΑΓ͸׳Ε͍͔ͯͨΒ • Google Cloud Platform΋࢖͍͔ͬͯͨΒ • ͪͳΈʹࣅͨΑ͏ͳπʔϧʹ͜Μͳͷ΋͋Γ·͢ • Apache Libcloud • https://libcloud.apache.org/index.html • छʑͷΫϥ΢υʹରԠͨ͠boto(aws sdk for Python)ͷ Α͏ͳ΋ͷ

Slide 23

Slide 23 text

؊UGTUBUFϑΝΠϧͷѻ͍ • terraform࣮ߦޙͷঢ়ଶ͕هड़͞Ε͍ͯΔϑΝΠϧ • terraform͸͜ͷϑΝΠϧͷঢ়ଶΛਖ਼ͱͯ͠มߋΛద༻͢Δ • terraform࣮ߦ࣌ʹ͜ͷϑΝΠϧ͕յΕͨΓઌ૆ฦΓͨ͠ঢ়ଶͰ࣮ߦ͢Δͱ ࣄނΔ • ͜ͷϑΝΠϧ͕յΕͨΒʁר͖໭ͬͯ͠·ͬͨΒʁ => ؤு࣏ͬͯͦ ͏:(ʀƄƅ’Тƅ'): • खݩ͔Β࣮ߦ͢Δ৔߹͸githubͳͲͰ؅ཧͭͭ͠࡞ۀऀ֤Ґ͕ඞͣ࠷৽ͷ ΋ͷΛpull͔ͯ͠Β࢖͏ • τϨλͰ͸… • S3ʹஔ͖ͭͭɺCircle CI͔ΒterraformΛ࣮ߦ • ΦϖϨʔγϣϯΛ࣮ߦ͢Δਓ෺͸Ұਓ(Circle CI)ʹ͢ΔͨΊ • ඞͣ࠷৽ͷtfstateϑΝΠϧΛ࢖͏Α͏ͳ࢓૊Έʹ͢ΔͨΊ

Slide 24

Slide 24 text

؊ESZSVO͕௨͔ͬͨΒͱ͍ͬͯʜ • ࣮ߦ͕੒ޭ͢Δͱ͸ݶΒͳ͍ • ͜ͷ৔߹͸མͪண͍ͯΤϥʔ಺༰ΛݟΔ • τϨλͰ͸AWSϦιʔεͷ্ݶʹୡͨ͠৔߹ͳͲʹ௚໘ ͨ͠(EC2Πϯελϯε਺ͷ্ݶ…etc)

Slide 25

Slide 25 text

؊͢΂ͯΛUFSSBGPSN؅ཧԼʹ͸ஔ͔ͳ͍ • tfstateϑΝΠϧΛਖ਼ͱͯ͠ಈ͘ͷͰঢ়ଶ͕มԽ͢ΔASG ͳͲʹ͸ෆ޲͖(ͩͱࢥ͍ͬͯ·͢) • ͦΕͱ…ͩͬͯා͍Μͩ΋Μ

Slide 26

Slide 26 text

࣮ࡍੲ͸݁ߏා͔ͬͨ • ͔ͭͯ͸༧ظͤ͵ഁյతͳڍಈ͕͋ͬͨ

Slide 27

Slide 27 text

3PBEXPSLFS

Slide 28

Slide 28 text

3PBEXPSLFS • https://github.com/winebarrel/ • Route53ͷϨίʔυઃఆΛRubyͷDSLͰهड़Ͱ͖Δ

Slide 29

Slide 29 text

3PBEXPSLFS • Engineyard࣌୅͸EngineyardͱAWSͷϋΠϒϦουӡ༻ • Engineyard: EC2Πϯελϯε(App, Redis, MySQL…) • AWS: Route53, S3… • ೖࣾ࣌ʹAWS্ͷ΋ͷͷίʔυԽʹணखͨ͠ • ࠷ॳ͔ΒTerraformʹ͠ͳ͔ͬͨͷ͸౰࣌ͷTerraform͸ طଘͷDNSϨίʔυͷexport͕Ͱ͖ͳ͔ͬͨ • Roadworker͸Ͱ͖ͨ

Slide 30

Slide 30 text

1BDLFS

Slide 31

Slide 31 text

1BDLFS • https://www.packer.io/ • Hashicorp੡ϓϩμΫτ • AMI(AWS), Virtualbox, dockerͳͲ֤छΠϝʔδΛϓϩά ϥϚϒϧʹ࡞੒Ͱ͖Δ • jsonͰهड़

Slide 32

Slide 32 text

1BDLFS • τϨλͰͷ༻్ • ϕʔεͱͳΔAMIͷ࡞੒ • ։ൃ؀ڥ༻Vagrant boxͷ࡞੒ • Packerʹ͍ͭͯ͸͜Εʹ୅ΘΔ΋ͷ͸ແ͍(͸ͣ)

Slide 33

Slide 33 text

1BDLFSͰ΍͍ͬͯΔ͜ͱ • OSͷॳظઃఆ • RubyͷΠϯετʔϧ • ϛυϧ΢ΣΞͷΠϯετʔϧ΍ίϯϑΟάͷ഑෍͸ Ansibleʹ΍Β͍ͤͯΔ • Ҏલ͸ϛυϧ΢ΣΞͷΠϯετʔϧ΋Packerʹ΍Β ͤͯશ෦ೖΓͷAMIΛ࡞͍ͬͯͨ(ϒϩάهࣄΑΓ↓)

Slide 34

Slide 34 text

શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ • શ෦ೖΓʹͨ͠৔߹ͷ… • ϝϦοτ • ىಈ͢Δ͚ͩͰαʔϏε౤ೖՄೳͱͳΔ • σϝϦοτ • ίϯϑΟάมߋͷͨͼʹAMIΛࣽࠐΉඞཁ͕͋Δ • ·ͨͦͷͨͼʹΠϯελϯεશೖΕସ͑ • ͜ͷ࢓૊ΈΛࣗಈԽ͢Ε͹σϝϦοτ͸ͳ͘ͳΔ͔΋͠Εͳ͍ • ͨͩࣽ͜͠Ή࣌ؒΛ଴ͭͷ͕μϧ͍ͱ͖΋͋Δ…

Slide 35

Slide 35 text

શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ • શ෦ೖΓͷํ͕Ϋϥ΢υతͰ͸͋Δ • ίϯςφͷӡ༻ͱۙ͘ͳΔ͸ͣ • ͭ·Γঢ়ଶͷมߋ = ΠϯελϯεΛ৽͘͠࡞ΔˍೖΕସ͑Δ • ίϯϑΟάྨͷมߋΛAnsible/chefͳͲͰద༻Λ܁Γฦ͢ͷ͸ݩ ͸ΦϯϓϨͷߟ͑ํ • ͭ·Γಉ͡αʔόΛ࢖͍ճ͢ͱ͍͏લఏͷιϦϡʔγϣϯ • ͍ΘΏΔႈ౳ੑ • Ϋϥ΢υ͸”ࣺͯΔ”ͱ͍͏બ୒ࢶ͕͋Δ • ࣺͯΔ->࠶ߏங͕୹࣌ؒͰՄೳ • ΋ͪΖΜΠϯελϯε͕յΕΔ·Ͱ࢖͍ճ͢͜ͱ΋Ͱ͖Δ

Slide 36

Slide 36 text

"OTJCMF

Slide 37

Slide 37 text

"OTJCMF • https://www.ansible.com/ • αʔόߏஙͷࣗಈԽɺႈ౳ੑͷ୲อ • ಉ౳ͷπʔϧʹchef΍itamae΍puppet͕͋Δ • yamlͰهड़

Slide 38

Slide 38 text

ͳͥ"OTJCMF͔ʁ • ผʹͳΜͰ΋ྑ͔ͬͨ • Α͋͘Δ(?)ࣄҊ • αʔόߏஙεΫϦϓτ͕ൿ఻ͷλϨͩᵆ( :ô^)ᵒ • αʔόߏஙखॱॻ͕ոจॻͩᵎ(ô^; )ᵊ • chef/ansibleΛಋೖͩ(^q^) • cookbook/playbook͕ݹจॻʹͳͬͨʗ(ô^)ʘ • ͜ͷखͷπʔϧ͸ͲΕ࢖ͬͯ΋هड़ϧʔϧ΍ӡ༻ϧʔϧΛܾΊ͓͔ͯͳ͍ ͱυπϘʹϋϚΔ(ܦݧ্) • Πϯϑϥίʔυ΋ͪΌΜͱϨϏϡʔ͢Δʢ͋ͨΓ·͑Ͱ͕͢…ʣ • ςετɺServerspecΛॻ͍͓ͯ͜͏ • ʮ༨ܭͳ͜ͱ͕Ͱ͖ͳ͍puppet͕Ұ൪ྑ͍ʯbyಉۀऀͷ༑ਓ ͱ͍͏ҙݟ΋ ͋Δ

Slide 39

Slide 39 text

"OTJCMF • τϨλͰ͸… • EC2ʹରͯ͠ϛυϧ΢ΣΞͷΠϯετʔϧɺίϯϑΟάͷ ഑෍Λߦ͍ͬͯΔ • μΠφϛοΫΠϯϕϯτϦΛར༻ • EC2ʹ෇༩ͨ͠ServiceλάͱRoleλάͰϓϩϏδϣχ ϯάର৅ͷϗετͱϩʔϧΛಈతʹऔಘ • ϗετͷ૿ݮɺIPͷมԽΛҙࣝͤͣʹϓϩϏδϣχϯ άͰ͖Δ

Slide 40

Slide 40 text

4FSWFSTQFD

Slide 41

Slide 41 text

4FSWFSTQFD • http://serverspec.org/ • ίϯϑΟάͷςετϑϨʔϜϫʔΫ • ϦϑΝΫλϦϯά͕ओ໨తͷιϑτ΢ΣΞ • RSpecͰهड़

Slide 42

Slide 42 text

4FSWFSTQFD • τϨλͰ΋ΠϯϑϥίʔυΛϦϑΝΫλϦϯά͠΍ ͘͢͢ΔͨΊʹॻ͍͍ͯΔ • Serverspec͕ॻ͍ͯ͋Ε͹ྫ͑͹ansible->chef΁ͷ ৐Γ׵͑(ͨͿΜ΍Βͳ͍͚Ͳ)΋΍Γ΍͍͢

Slide 43

Slide 43 text

"VSPSB

Slide 44

Slide 44 text

"VSPSB • RDSͷΤϯδϯͷҰͭ • MySQLޓ׵ • GUIϙνϙνͰߏஙͯ͠·͢ • ↓ͱ͍͏ߟ͑(ݴ͍༁ʁ)

Slide 45

Slide 45 text

"VSPSB • ಋೖʹ͋ͨͬͯݕূͨ͜͠ͱ • Ҡߦલ(MySQL on Engineyard)ͱ࣮ߦܭը͕มΘΒͳ͍͜ͱͷ֬ೝ • ͪΌΜͱࠓ·Ͱ௨ΓʹΠϯσοΫεΛ࢖ͬͯ͘ΕΔ͔ • ίϯϑΟάϨʔγϣϯͷਫ਼ࠪ • MySQL͓͡͞Μͱͯ͠͸ඇৗʹؾʹͳͬͨ… • Ͳ͜ΛͲ͏มߋ͔ͨ͠͸ϒϩάهࣄΛࢀর͍ͯͩ͘͠͞ • εέʔϧΞοϓɺσʔλϦΧόϦ…ͳͲͷΦϖखॱɺཁ͢Δ࣌ؒͳͲ • ༗ࣄʹඋ͑ͯखॱΛཱ͓֬ͯͨ͘͠Ί • ཁ͸ૉৼΓɻ࿅शɻ

Slide 46

Slide 46 text

"VSPSB • ੑೳࢼݧ͸΍ͬͯͳ͍ɻͱ͍͏͔΍Βͳ͍ɻ • sysbenchͳͲͷࢼݧʹ͍ͭͯ͸͢Ͱʹଟ͘ͷهࣄ͕ωο τʹ্͕͍ͬͯΔɻਓͱಉ͜͡ͱΛͯ͠΋ҙຯ͕ͳ͍ɻ • ΋͠΍ΔͳΒຊ൪૬౰ͷσʔλྔɺSQLɺτϥϑΟοΫ • όονॲཧͷࢀরܥΛAuroraʹ޲͚ͯෛՙঢ়گΛ֬ೝ • ΦϯϥΠϯॲཧʹ͍ͭͯ͸લεϥΠυͷ௨Γ࣮ߦܭըͷ ֬ೝΛߦͬͯMySQLͱಉ౳Ͱ͋Δ͜ͱΛ֬ೝͯ͠OKͱ͠ ͨ

Slide 47

Slide 47 text

"VSPSBͰ͸࣮ݱͰ͖ͳ͍͜ͱ • θϩμ΢ϯλΠϜ • ϝϯςφϯε΢Οϯυ΢͕͋ΔͷͰ͍͔ͭ͸࠶ىಈ͠ͳ͖Ό ͳΒͳ͍ͱ͖͕དྷΔɻ • Ͱ΋ϑΣΠϧΦʔόͰ1෼ఔ౓Ͱ׬ • 1෼΋ࢭΊͨ͘ͳ͍৔߹͸… • MySQL on EC2 with MHA • ࢲͷهԱͰ͸MHAͷํ͕ϑΣΠϧΦʔό͸ૣ͍ • ׬શθϩμ΢ϯλΠϜΛ໨ࢦ͔ͨͬͨ͠ΒϚϧνϚελͳ෼ ࢄDBͰ΋…Cassandraͱ͔Ͱ…ؤுͬͯ…͍ͩ͘͞…

Slide 48

Slide 48 text

$JSDMF$*

Slide 49

Slide 49 text

$JSDMF$* • ஶ໊ͳCIαʔϏεͷҰͭ • RailsΞϓϦέʔγϣϯͷCI/σϓϩΠج൫ͱͯ͠׆༻ • ΠϯϑϥΦϖϨʔγϣϯͷ࣮ߦج൫ͱͯ͠΋׆༻

Slide 50

Slide 50 text

$JSDMF$*͔ΒͷσϓϩΠ • EC2ʹ෇༩͞ΕͨServiceλάͱRoleλάͰର৅ͷϗ ετΛಛఆͯ͠Capistrano΍AnsibleΛ࣮ߦ • ·͋ී௨ͷ΍ΓํͰ͢ • θϩμ΢ϯλΠϜσϓϩΠͷ࣮ݱ͸ʁ • unicorn͸ϗοτσϓϩΠ͕ޮ͘ • ࠓͷن໛Ͱ͸͜ΕͰे෼

Slide 51

Slide 51 text

ࠓޙ

Slide 52

Slide 52 text

ࠓޙ • ίϯςφʁ • ΍ΔͳΒ৽نαʔϏε։ൃ࣌ • ։ൃख๏ɺσϓϩΠํ๏ɺӡ༻ख๏͕มԽ͢ΔɺଞΤϯδχΞʹ΋ͦΕΛ΍ͬ ͯ΋Β͏ඞཁ͕͋Δ • طଘͷαʔϏεΛίϯςφԽ͢Δ͜ͱ͸ࠓ͸ߟ͍͑ͯͳ͍ • ৽نαʔϏεͰίϯςφಋೖͷޮՌ͕ग़ͨΒ΍Δ͔΋ • ASGʁ • ୆਺͕૿͑ͯඇϐʔΫ࣌ͷίετ࡟ݮޮՌ͕ݟ͑ͦ͏ʹͳͬͨΒ • ͨͩ͠Terraformͱͷ૬ੑ͕ѱͦ͏ͩ

Slide 53

Slide 53 text

͓ΘΓ