トレタのインフラ運用

【トレタ×プレイド】TechBlog Deep Dive Meetup #1 〜インフラ編〜
http://plaidtech.connpass.com/event/33511/

での発表内容。
Terraform, Packer, Roadworker, Ansible, Serverspec, Circle CIなどをトレタでどう使っているか？何が罠か？など。

Transcript

1. τϨλͷΠϯϑϥӡ༻ ʲτϨλºϓϨΠυʳ5FDI#MPH
   %FFQ
   %JWF
   .FFUVQ
   
   
   )JSPBLJ
   4BOP
2. NF • Name: • ࠤ໺༟ষ(Hiroaki Sano) • Personal website: • https://hiroakis.com/blog/ • Company: • NEC Soft, Ltd. (2006/4-) • CyberAgent, Inc. (2011/3-) • Toreta, Inc. (2014/11-)
5. τϨλ • ҿ৯ళ޲͚༧໿؅ཧΞϓϦέʔγϣϯΛiPadΞϓϦͱͯ͠ఏڙ • ҰൠϢʔβ޲͚ʹ΢Σϒ༧໿΋ఏڙ • ސ٬ɿϨετϥϯɺډञ԰ͳͲͷҿ৯ళ • ྫ͑͹… • Զͷגࣜձ༷ࣾ(ԶͷΠλϦΞϯɺԶͷϑϨϯν…etc) • גࣜձࣾେঙ༷(ঙ΍ɺ΍Δ͖஡԰…etc) • ΤʔɾϐʔΧϯύχʔ༷(௩ా೶৔…etc) • ϩʔετϗʔε༷ • etc • ϩʔϯν͸2013೥12݄ • ಋೖళฮ਺ɿ6000ళฮʙ • ళฮ͋ͨΓͷֹ݄՝ۚ • ίʔϙϨʔτΧϥʔ͸྘ • ΦϑΟε͸ौ୩ -> ໨ࠇ -> ܙൺण -> ޒ൓ా
6. ͜ΕΛ CONFIDENTIAL - Toreta Inc., All Right Reserved. Toreta Inc., All Right Reserved.
7. ͜͏ CONFIDENTIAL - Toreta Inc., All Right Reserved. Toreta Inc., All Right Reserved.
8. τϨλͷίϯηϓτ • ҿ৯ళͷ༧໿؅ཧɺސ٬؅ཧΛITԽ • ҿ৯ళͰಇ͘ਓͷۀ຿ͷࣗಈԽ • ༧໿ࣄނΛ๷͙
9. ࠓ೔͸ٕज़ϒϩάਂ۷Γձ ͱ͍͏͜ͱͰʜ
10. τϨλ։ൃऀϒϩά
    IUUQUFDIUPSFUBJO
12. ࢲͷॻ͍ͨهࣄ • Raspberry PiͱԹ౓ɾ࣪౓ɾޫηϯαʔͰΦϑΟεͷঢ়گΛՄࢹԽ͢Δ • http://tech.toreta.in/entry/2016/02/15/153203 • τϨλͷΠϯϑϥӡ༻ɺࢧ͍͑ͯΔಓ۩(Packer, Terraform, Serverspec, Ansible, Roadworker, Circle CI)ɺߟ͑ํ • http://tech.toreta.in/entry/2016/04/14/143248 • ϝʔϧͷ഑৴ঢ়گΛՄࢹԽɺ௥੻͢Δ • http://tech.toreta.in/entry/2016/04/20/153852 • Engineyard͔ΒAWSʹҠઃͯ͠Auroraͷӡ༻Λ։࢝ͨ͠ • http://tech.toreta.in/entry/2016/06/16/114919
13. ࠓ೔࿩͢͜ͱ • ओʹAWSͰͷΠϯϑϥӡ༻ʹ͍ͭͯ • τϨλͷAPIαʔόͷΤίγεςϜ • ओʹϒϩάωλʹొ৔ͨ͠ࣄฑΛத৺ʹ… • Terraform • Roadworker • Packer • Ansible • Serverspec • Aurora • Circle CI • ࠓޙʹ͍ͭͯ • ٕज़తʹਅ৽͍͠΋ͷ͸ͳ͍Ͱ͕͢ࢀߟʹ͍͚ͯͨͩ͠Ε͹…
14. τϨλ"1*ͱͦͷपลͷΤίγεςϜ
15. Amazon Route 53 Amazon RDS(Aurora) api
 worker Redis memcached Amazon SNS Amazon SQS τϨλ"1*ͱͦͷपลͷΤίγεςϜ FAX/SMS… Amazon S3 ը૾/Ի੠… ଞαϒγεςϜ΁ͷ௨஌ ϩά ϝʔϧ fax, sms API worker fluentd
16. τϨλ"1*ͱͦͷपลͷΤίγεςϜ • Amazon Web Services • 5݄ʹEngineyard͔ΒҠઃ(ϒϩάهࣄͷ௨Γ) • Appαʔό: APIɺWorkerಉډ. nginx, Rails(unicorn), sidekiqͰ࣮૷. • Redis: WorkerͷΩϡʔΠϯά༻ • memcached: Ωϟογϡ • RDS Aurora: ϝΠϯͷσʔλϕʔε • Google Cloud Platform • BigQuery: fluentd͔Βϩάͷసૹ • ιʔϦʔαʔό΍Ұ෦ͷόοναʔό: GCEͷݕূΛ݉Ͷͯͪ͜Βʹߏஙͯ͋͠Δ • Monitoring • Mackerel: Ϧιʔε؂ࢹɺϝτϦΫε • Pingdom: ΤϯυϙΠϯτ؂ࢹ • Pagerduty: ΞϥʔτରԠͷεέδϡʔϦϯάɺి࿩௨஌ • CI • Circle CI: CI͓ΑͼσϓϩΠɺ͓ΑͼΦϖϨʔγϣϯͷத৺
17. app
 worker Redis Aurora Writer memcached app
 worker Redis memcached Availability Zone A 71$ઃܭ public subnet Availability Zone C Aurora Reader public subnet 10.0.0.0/16 10.0.0.0/24 10.0.1.0/24
18. 71$ઃܭ • ֤ίϯϙʔωϯτ͸ϚϧνAZͰ഑ஔ • 10.0.X.0/24 -> AZ-A • 10.0.Y.0/24 -> AZ-C • ύϒϦοΫαϒωοτͷΈ • RDSͳͲ΋ύϒϦοΫαϒωοτʹஔ͘ • ݎ࿚ੑ͸ηΩϡϦςΟάϧʔϓͰ୲อ • ࡶ͡Όͳ͍͔ʁͱͨ·ʹݴΘΕΔ • ݱঢ়ͷ࡞ΓͰผʹࠔ͍ͬͯͳ͍ • ࠔͬͨΒ࡞Γͳ͓͢
19. 5FSSBGPSN
20. 5FSSBGPSN • https://www.terraform.io/ • Hashicorp੡ϓϩμΫτ • AWS΍GCPͳͲ֤छΫϥ΢υͷঢ়ଶΛίʔυͰهड़͢Δ΋ͷ • HCLͱ͍͏jsonϥΠΫͳϑΥʔϚοτͰهड़ ग़య: https://www.terraform.io/
21. 5FSSBGPSN • VPC, SecurityGroup, EC2Λ؅ཧର৅ͱ͍ͯ͠Δ • EC2ߏங࣌͸Environment, Role, ServiceλάΛ෇༩͢Δ • ͜ΕΒ͸σϓϩΠ΍ϓϩϏδϣχϯάͰ࢖͏ • AWSͷ࢖͍ํͱͯ͠యܕతͱ͍͏͔ݹయతͳखஈ
22. ͳͥ5FSSBGPSN͔ʁ • CloudFormationΑΓ͸׳Ε͍͔ͯͨΒ • Google Cloud Platform΋࢖͍͔ͬͯͨΒ • ͪͳΈʹࣅͨΑ͏ͳπʔϧʹ͜Μͳͷ΋͋Γ·͢ • Apache Libcloud • https://libcloud.apache.org/index.html • छʑͷΫϥ΢υʹରԠͨ͠boto(aws sdk for Python)ͷ Α͏ͳ΋ͷ
23. ؊UGTUBUFϑΝΠϧͷѻ͍ • terraform࣮ߦޙͷঢ়ଶ͕هड़͞Ε͍ͯΔϑΝΠϧ • terraform͸͜ͷϑΝΠϧͷঢ়ଶΛਖ਼ͱͯ͠มߋΛద༻͢Δ • terraform࣮ߦ࣌ʹ͜ͷϑΝΠϧ͕յΕͨΓઌ૆ฦΓͨ͠ঢ়ଶͰ࣮ߦ͢Δͱ ࣄނΔ • ͜ͷϑΝΠϧ͕յΕͨΒʁר͖໭ͬͯ͠·ͬͨΒʁ => ؤு࣏ͬͯͦ ͏:(ʀƄƅ’Тƅ'): • खݩ͔Β࣮ߦ͢Δ৔߹͸githubͳͲͰ؅ཧͭͭ͠࡞ۀऀ֤Ґ͕ඞͣ࠷৽ͷ ΋ͷΛpull͔ͯ͠Β࢖͏ • τϨλͰ͸… • S3ʹஔ͖ͭͭɺCircle CI͔ΒterraformΛ࣮ߦ • ΦϖϨʔγϣϯΛ࣮ߦ͢Δਓ෺͸Ұਓ(Circle CI)ʹ͢ΔͨΊ • ඞͣ࠷৽ͷtfstateϑΝΠϧΛ࢖͏Α͏ͳ࢓૊Έʹ͢ΔͨΊ
24. ؊
    ESZSVO͕௨͔ͬͨΒͱ͍ͬͯʜ • ࣮ߦ͕੒ޭ͢Δͱ͸ݶΒͳ͍ • ͜ͷ৔߹͸མͪண͍ͯΤϥʔ಺༰ΛݟΔ • τϨλͰ͸AWSϦιʔεͷ্ݶʹୡͨ͠৔߹ͳͲʹ௚໘ ͨ͠(EC2Πϯελϯε਺ͷ্ݶ…etc)
25. ؊
    ͢΂ͯΛUFSSBGPSN؅ཧԼʹ͸ஔ͔ͳ͍ • tfstateϑΝΠϧΛਖ਼ͱͯ͠ಈ͘ͷͰঢ়ଶ͕มԽ͢ΔASG ͳͲʹ͸ෆ޲͖(ͩͱࢥ͍ͬͯ·͢) • ͦΕͱ…ͩͬͯා͍Μͩ΋Μ
26. ࣮ࡍੲ͸݁ߏා͔ͬͨ • ͔ͭͯ͸༧ظͤ͵ഁյతͳڍಈ͕͋ͬͨ
27. 3PBEXPSLFS
28. 3PBEXPSLFS • https://github.com/winebarrel/ • Route53ͷϨίʔυઃఆΛRubyͷDSLͰهड़Ͱ͖Δ
29. 3PBEXPSLFS • Engineyard࣌୅͸EngineyardͱAWSͷϋΠϒϦουӡ༻ • Engineyard: EC2Πϯελϯε(App, Redis, MySQL…) • AWS: Route53, S3… • ೖࣾ࣌ʹAWS্ͷ΋ͷͷίʔυԽʹணखͨ͠ • ࠷ॳ͔ΒTerraformʹ͠ͳ͔ͬͨͷ͸౰࣌ͷTerraform͸ طଘͷDNSϨίʔυͷexport͕Ͱ͖ͳ͔ͬͨ • Roadworker͸Ͱ͖ͨ
30. 1BDLFS
31. 1BDLFS • https://www.packer.io/ • Hashicorp੡ϓϩμΫτ • AMI(AWS), Virtualbox, dockerͳͲ֤छΠϝʔδΛϓϩά ϥϚϒϧʹ࡞੒Ͱ͖Δ • jsonͰهड़
32. 1BDLFS • τϨλͰͷ༻్ • ϕʔεͱͳΔAMIͷ࡞੒ • ։ൃ؀ڥ༻Vagrant boxͷ࡞੒ • Packerʹ͍ͭͯ͸͜Εʹ୅ΘΔ΋ͷ͸ແ͍(͸ͣ)
33. 1BDLFSͰ΍͍ͬͯΔ͜ͱ • OSͷॳظઃఆ • RubyͷΠϯετʔϧ • ϛυϧ΢ΣΞͷΠϯετʔϧ΍ίϯϑΟάͷ഑෍͸ Ansibleʹ΍Β͍ͤͯΔ • Ҏલ͸ϛυϧ΢ΣΞͷΠϯετʔϧ΋Packerʹ΍Β ͤͯશ෦ೖΓͷAMIΛ࡞͍ͬͯͨ(ϒϩάهࣄΑΓ↓)
34. શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ • શ෦ೖΓʹͨ͠৔߹ͷ… • ϝϦοτ • ىಈ͢Δ͚ͩͰαʔϏε౤ೖՄೳͱͳΔ • σϝϦοτ • ίϯϑΟάมߋͷͨͼʹAMIΛࣽࠐΉඞཁ͕͋Δ • ·ͨͦͷͨͼʹΠϯελϯεશೖΕସ͑ • ͜ͷ࢓૊ΈΛࣗಈԽ͢Ε͹σϝϦοτ͸ͳ͘ͳΔ͔΋͠Εͳ͍ • ͨͩࣽ͜͠Ή࣌ؒΛ଴ͭͷ͕μϧ͍ͱ͖΋͋Δ…
35. શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ • શ෦ೖΓͷํ͕Ϋϥ΢υతͰ͸͋Δ • ίϯςφͷӡ༻ͱۙ͘ͳΔ͸ͣ • ͭ·Γঢ়ଶͷมߋ = ΠϯελϯεΛ৽͘͠࡞ΔˍೖΕସ͑Δ • ίϯϑΟάྨͷมߋΛAnsible/chefͳͲͰద༻Λ܁Γฦ͢ͷ͸ݩ ͸ΦϯϓϨͷߟ͑ํ • ͭ·Γಉ͡αʔόΛ࢖͍ճ͢ͱ͍͏લఏͷιϦϡʔγϣϯ • ͍ΘΏΔႈ౳ੑ • Ϋϥ΢υ͸”ࣺͯΔ”ͱ͍͏બ୒ࢶ͕͋Δ • ࣺͯΔ->࠶ߏங͕୹࣌ؒͰՄೳ • ΋ͪΖΜΠϯελϯε͕յΕΔ·Ͱ࢖͍ճ͢͜ͱ΋Ͱ͖Δ
36. "OTJCMF
37. "OTJCMF • https://www.ansible.com/ • αʔόߏஙͷࣗಈԽɺႈ౳ੑͷ୲อ • ಉ౳ͷπʔϧʹchef΍itamae΍puppet͕͋Δ • yamlͰهड़
38. ͳͥ"OTJCMF͔ʁ • ผʹͳΜͰ΋ྑ͔ͬͨ • Α͋͘Δ(?)ࣄҊ • αʔόߏஙεΫϦϓτ͕ൿ఻ͷλϨͩᵆ( :^o^)ᵒ • αʔόߏஙखॱॻ͕ոจॻͩᵎ(^o^; )ᵊ • chef/ansibleΛಋೖͩ(^q^) • cookbook/playbook͕ݹจॻʹͳͬͨʗ(^o^)ʘ • ͜ͷखͷπʔϧ͸ͲΕ࢖ͬͯ΋هड़ϧʔϧ΍ӡ༻ϧʔϧΛܾΊ͓͔ͯͳ͍ ͱυπϘʹϋϚΔ(ܦݧ্) • Πϯϑϥίʔυ΋ͪΌΜͱϨϏϡʔ͢Δʢ͋ͨΓ·͑Ͱ͕͢…ʣ • ςετɺServerspecΛॻ͍͓ͯ͜͏ • ʮ༨ܭͳ͜ͱ͕Ͱ͖ͳ͍puppet͕Ұ൪ྑ͍ʯbyಉۀऀͷ༑ਓ ͱ͍͏ҙݟ΋ ͋Δ
39. "OTJCMF • τϨλͰ͸… • EC2ʹରͯ͠ϛυϧ΢ΣΞͷΠϯετʔϧɺίϯϑΟάͷ ഑෍Λߦ͍ͬͯΔ • μΠφϛοΫΠϯϕϯτϦΛར༻ • EC2ʹ෇༩ͨ͠ServiceλάͱRoleλάͰϓϩϏδϣχ ϯάର৅ͷϗετͱϩʔϧΛಈతʹऔಘ • ϗετͷ૿ݮɺIPͷมԽΛҙࣝͤͣʹϓϩϏδϣχϯ άͰ͖Δ
40. 4FSWFSTQFD
41. 4FSWFSTQFD • http://serverspec.org/ • ίϯϑΟάͷςετϑϨʔϜϫʔΫ • ϦϑΝΫλϦϯά͕ओ໨తͷιϑτ΢ΣΞ • RSpecͰهड़
42. 4FSWFSTQFD • τϨλͰ΋ΠϯϑϥίʔυΛϦϑΝΫλϦϯά͠΍ ͘͢͢ΔͨΊʹॻ͍͍ͯΔ • Serverspec͕ॻ͍ͯ͋Ε͹ྫ͑͹ansible->chef΁ͷ ৐Γ׵͑(ͨͿΜ΍Βͳ͍͚Ͳ)΋΍Γ΍͍͢
43. "VSPSB
44. "VSPSB • RDSͷΤϯδϯͷҰͭ • MySQLޓ׵ • GUIϙνϙνͰߏஙͯ͠·͢ • ↓ͱ͍͏ߟ͑(ݴ͍༁ʁ)
45. "VSPSB • ಋೖʹ͋ͨͬͯݕূͨ͜͠ͱ • Ҡߦલ(MySQL on Engineyard)ͱ࣮ߦܭը͕มΘΒͳ͍͜ͱͷ֬ೝ • ͪΌΜͱࠓ·Ͱ௨ΓʹΠϯσοΫεΛ࢖ͬͯ͘ΕΔ͔ • ίϯϑΟάϨʔγϣϯͷਫ਼ࠪ • MySQL͓͡͞Μͱͯ͠͸ඇৗʹؾʹͳͬͨ… • Ͳ͜ΛͲ͏มߋ͔ͨ͠͸ϒϩάهࣄΛࢀর͍ͯͩ͘͠͞ • εέʔϧΞοϓɺσʔλϦΧόϦ…ͳͲͷΦϖखॱɺཁ͢Δ࣌ؒͳͲ • ༗ࣄʹඋ͑ͯखॱΛཱ͓֬ͯͨ͘͠Ί • ཁ͸ૉৼΓɻ࿅शɻ
46. "VSPSB • ੑೳࢼݧ͸΍ͬͯͳ͍ɻͱ͍͏͔΍Βͳ͍ɻ • sysbenchͳͲͷࢼݧʹ͍ͭͯ͸͢Ͱʹଟ͘ͷهࣄ͕ωο τʹ্͕͍ͬͯΔɻਓͱಉ͜͡ͱΛͯ͠΋ҙຯ͕ͳ͍ɻ • ΋͠΍ΔͳΒຊ൪૬౰ͷσʔλྔɺSQLɺτϥϑΟοΫ • όονॲཧͷࢀরܥΛAuroraʹ޲͚ͯෛՙঢ়گΛ֬ೝ • ΦϯϥΠϯॲཧʹ͍ͭͯ͸લεϥΠυͷ௨Γ࣮ߦܭըͷ ֬ೝΛߦͬͯMySQLͱಉ౳Ͱ͋Δ͜ͱΛ֬ೝͯ͠OKͱ͠ ͨ
47. "VSPSBͰ͸࣮ݱͰ͖ͳ͍͜ͱ • θϩμ΢ϯλΠϜ • ϝϯςφϯε΢Οϯυ΢͕͋ΔͷͰ͍͔ͭ͸࠶ىಈ͠ͳ͖Ό ͳΒͳ͍ͱ͖͕དྷΔɻ • Ͱ΋ϑΣΠϧΦʔόͰ1෼ఔ౓Ͱ׬ • 1෼΋ࢭΊͨ͘ͳ͍৔߹͸… • MySQL on EC2 with MHA • ࢲͷهԱͰ͸MHAͷํ͕ϑΣΠϧΦʔό͸ૣ͍ • ׬શθϩμ΢ϯλΠϜΛ໨ࢦ͔ͨͬͨ͠ΒϚϧνϚελͳ෼ ࢄDBͰ΋…Cassandraͱ͔Ͱ…ؤுͬͯ…͍ͩ͘͞…
48. $JSDMF
    $*
49. $JSDMF
    $* • ஶ໊ͳCIαʔϏεͷҰͭ • RailsΞϓϦέʔγϣϯͷCI/σϓϩΠج൫ͱͯ͠׆༻ • ΠϯϑϥΦϖϨʔγϣϯͷ࣮ߦج൫ͱͯ͠΋׆༻
50. $JSDMF
    $*͔ΒͷσϓϩΠ • EC2ʹ෇༩͞ΕͨServiceλάͱRoleλάͰର৅ͷϗ ετΛಛఆͯ͠Capistrano΍AnsibleΛ࣮ߦ • ·͋ී௨ͷ΍ΓํͰ͢ • θϩμ΢ϯλΠϜσϓϩΠͷ࣮ݱ͸ʁ • unicorn͸ϗοτσϓϩΠ͕ޮ͘ • ࠓͷن໛Ͱ͸͜ΕͰे෼
51. ࠓޙ
52. ࠓޙ • ίϯςφʁ • ΍ΔͳΒ৽نαʔϏε։ൃ࣌ • ։ൃख๏ɺσϓϩΠํ๏ɺӡ༻ख๏͕มԽ͢ΔɺଞΤϯδχΞʹ΋ͦΕΛ΍ͬ ͯ΋Β͏ඞཁ͕͋Δ • طଘͷαʔϏεΛίϯςφԽ͢Δ͜ͱ͸ࠓ͸ߟ͍͑ͯͳ͍ • ৽نαʔϏεͰίϯςφಋೖͷޮՌ͕ग़ͨΒ΍Δ͔΋ • ASGʁ • ୆਺͕૿͑ͯඇϐʔΫ࣌ͷίετ࡟ݮޮՌ͕ݟ͑ͦ͏ʹͳͬͨΒ • ͨͩ͠Terraformͱͷ૬ੑ͕ѱͦ͏ͩ
53. ͓ΘΓ