Upgrade to Pro — share decks privately, control downloads, hide ads and more …

トレタのインフラ運用

 トレタのインフラ運用

【トレタ×プレイド】TechBlog Deep Dive Meetup #1 〜インフラ編〜
http://plaidtech.connpass.com/event/33511/

での発表内容。
Terraform, Packer, Roadworker, Ansible, Serverspec, Circle CIなどをトレタでどう使っているか?何が罠か?など。

Db2247a5b4226e1f6f74a9bd87810090?s=128

Hiroaki Sano

July 06, 2016
Tweet

Transcript

  1. τϨλͷΠϯϑϥӡ༻ ʲτϨλºϓϨΠυʳ5FDI#MPH%FFQ%JWF.FFUVQ  )JSPBLJ4BOP

  2. NF • Name: • ࠤ໺༟ষ(Hiroaki Sano) • Personal website: •

    https://hiroakis.com/blog/ • Company: • NEC Soft, Ltd. (2006/4-) • CyberAgent, Inc. (2011/3-) • Toreta, Inc. (2014/11-)
  3. None
  4. None
  5. τϨλ • ҿ৯ళ޲͚༧໿؅ཧΞϓϦέʔγϣϯΛiPadΞϓϦͱͯ͠ఏڙ • ҰൠϢʔβ޲͚ʹ΢Σϒ༧໿΋ఏڙ • ސ٬ɿϨετϥϯɺډञ԰ͳͲͷҿ৯ళ • ྫ͑͹… •

    Զͷגࣜձ༷ࣾ(ԶͷΠλϦΞϯɺԶͷϑϨϯν…etc) • גࣜձࣾେঙ༷(ঙ΍ɺ΍Δ͖஡԰…etc) • ΤʔɾϐʔΧϯύχʔ༷(௩ా೶৔…etc) • ϩʔετϗʔε༷ • etc • ϩʔϯν͸2013೥12݄ • ಋೖళฮ਺ɿ6000ళฮʙ • ళฮ͋ͨΓͷֹ݄՝ۚ • ίʔϙϨʔτΧϥʔ͸྘ • ΦϑΟε͸ौ୩ -> ໨ࠇ -> ܙൺण -> ޒ൓ా
  6. ͜ΕΛ CONFIDENTIAL - Toreta Inc., All Right Reserved. Toreta Inc.,

    All Right Reserved.
  7. ͜͏ CONFIDENTIAL - Toreta Inc., All Right Reserved. Toreta Inc.,

    All Right Reserved.
  8. τϨλͷίϯηϓτ • ҿ৯ళͷ༧໿؅ཧɺސ٬؅ཧΛITԽ • ҿ৯ళͰಇ͘ਓͷۀ຿ͷࣗಈԽ • ༧໿ࣄނΛ๷͙

  9. ࠓ೔͸ٕज़ϒϩάਂ۷Γձ ͱ͍͏͜ͱͰʜ

  10. τϨλ։ൃऀϒϩά IUUQUFDIUPSFUBJO

  11. None
  12. ࢲͷॻ͍ͨهࣄ • Raspberry PiͱԹ౓ɾ࣪౓ɾޫηϯαʔͰΦϑΟεͷঢ়گΛՄࢹԽ͢Δ • http://tech.toreta.in/entry/2016/02/15/153203 • τϨλͷΠϯϑϥӡ༻ɺࢧ͍͑ͯΔಓ۩(Packer, Terraform, Serverspec,

    Ansible, Roadworker, Circle CI)ɺߟ͑ํ • http://tech.toreta.in/entry/2016/04/14/143248 • ϝʔϧͷ഑৴ঢ়گΛՄࢹԽɺ௥੻͢Δ • http://tech.toreta.in/entry/2016/04/20/153852 • Engineyard͔ΒAWSʹҠઃͯ͠Auroraͷӡ༻Λ։࢝ͨ͠ • http://tech.toreta.in/entry/2016/06/16/114919
  13. ࠓ೔࿩͢͜ͱ • ओʹAWSͰͷΠϯϑϥӡ༻ʹ͍ͭͯ • τϨλͷAPIαʔόͷΤίγεςϜ • ओʹϒϩάωλʹొ৔ͨ͠ࣄฑΛத৺ʹ… • Terraform •

    Roadworker • Packer • Ansible • Serverspec • Aurora • Circle CI • ࠓޙʹ͍ͭͯ • ٕज़తʹਅ৽͍͠΋ͷ͸ͳ͍Ͱ͕͢ࢀߟʹ͍͚ͯͨͩ͠Ε͹…
  14. τϨλ"1*ͱͦͷपลͷΤίγεςϜ

  15. Amazon Route 53 Amazon RDS(Aurora) api
 worker Redis memcached Amazon

    SNS Amazon SQS τϨλ"1*ͱͦͷपลͷΤίγεςϜ FAX/SMS… Amazon S3 ը૾/Ի੠… ଞαϒγεςϜ΁ͷ௨஌ ϩά ϝʔϧ fax, sms API worker fluentd
  16. τϨλ"1*ͱͦͷपลͷΤίγεςϜ • Amazon Web Services • 5݄ʹEngineyard͔ΒҠઃ(ϒϩάهࣄͷ௨Γ) • Appαʔό: APIɺWorkerಉډ.

    nginx, Rails(unicorn), sidekiqͰ࣮૷. • Redis: WorkerͷΩϡʔΠϯά༻ • memcached: Ωϟογϡ • RDS Aurora: ϝΠϯͷσʔλϕʔε • Google Cloud Platform • BigQuery: fluentd͔Βϩάͷసૹ • ιʔϦʔαʔό΍Ұ෦ͷόοναʔό: GCEͷݕূΛ݉Ͷͯͪ͜Βʹߏஙͯ͋͠Δ • Monitoring • Mackerel: Ϧιʔε؂ࢹɺϝτϦΫε • Pingdom: ΤϯυϙΠϯτ؂ࢹ • Pagerduty: ΞϥʔτରԠͷεέδϡʔϦϯάɺి࿩௨஌ • CI • Circle CI: CI͓ΑͼσϓϩΠɺ͓ΑͼΦϖϨʔγϣϯͷத৺
  17. app
 worker Redis Aurora Writer memcached app
 worker Redis memcached

    Availability Zone A 71$ઃܭ public subnet Availability Zone C Aurora Reader public subnet 10.0.0.0/16 10.0.0.0/24 10.0.1.0/24
  18. 71$ઃܭ • ֤ίϯϙʔωϯτ͸ϚϧνAZͰ഑ஔ • 10.0.X.0/24 -> AZ-A • 10.0.Y.0/24 ->

    AZ-C • ύϒϦοΫαϒωοτͷΈ • RDSͳͲ΋ύϒϦοΫαϒωοτʹஔ͘ • ݎ࿚ੑ͸ηΩϡϦςΟάϧʔϓͰ୲อ • ࡶ͡Όͳ͍͔ʁͱͨ·ʹݴΘΕΔ • ݱঢ়ͷ࡞ΓͰผʹࠔ͍ͬͯͳ͍ • ࠔͬͨΒ࡞Γͳ͓͢
  19. 5FSSBGPSN

  20. 5FSSBGPSN • https://www.terraform.io/ • Hashicorp੡ϓϩμΫτ • AWS΍GCPͳͲ֤छΫϥ΢υͷঢ়ଶΛίʔυͰهड़͢Δ΋ͷ • HCLͱ͍͏jsonϥΠΫͳϑΥʔϚοτͰهड़ ग़య:

    https://www.terraform.io/
  21. 5FSSBGPSN • VPC, SecurityGroup, EC2Λ؅ཧର৅ͱ͍ͯ͠Δ • EC2ߏங࣌͸Environment, Role, ServiceλάΛ෇༩͢Δ •

    ͜ΕΒ͸σϓϩΠ΍ϓϩϏδϣχϯάͰ࢖͏ • AWSͷ࢖͍ํͱͯ͠యܕతͱ͍͏͔ݹయతͳखஈ
  22. ͳͥ5FSSBGPSN͔ʁ • CloudFormationΑΓ͸׳Ε͍͔ͯͨΒ • Google Cloud Platform΋࢖͍͔ͬͯͨΒ • ͪͳΈʹࣅͨΑ͏ͳπʔϧʹ͜Μͳͷ΋͋Γ·͢ •

    Apache Libcloud • https://libcloud.apache.org/index.html • छʑͷΫϥ΢υʹରԠͨ͠boto(aws sdk for Python)ͷ Α͏ͳ΋ͷ
  23. ؊UGTUBUFϑΝΠϧͷѻ͍ • terraform࣮ߦޙͷঢ়ଶ͕هड़͞Ε͍ͯΔϑΝΠϧ • terraform͸͜ͷϑΝΠϧͷঢ়ଶΛਖ਼ͱͯ͠มߋΛద༻͢Δ • terraform࣮ߦ࣌ʹ͜ͷϑΝΠϧ͕յΕͨΓઌ૆ฦΓͨ͠ঢ়ଶͰ࣮ߦ͢Δͱ ࣄނΔ • ͜ͷϑΝΠϧ͕յΕͨΒʁר͖໭ͬͯ͠·ͬͨΒʁ

    => ؤு࣏ͬͯͦ ͏:(ʀƄƅ’Тƅ'): • खݩ͔Β࣮ߦ͢Δ৔߹͸githubͳͲͰ؅ཧͭͭ͠࡞ۀऀ֤Ґ͕ඞͣ࠷৽ͷ ΋ͷΛpull͔ͯ͠Β࢖͏ • τϨλͰ͸… • S3ʹஔ͖ͭͭɺCircle CI͔ΒterraformΛ࣮ߦ • ΦϖϨʔγϣϯΛ࣮ߦ͢Δਓ෺͸Ұਓ(Circle CI)ʹ͢ΔͨΊ • ඞͣ࠷৽ͷtfstateϑΝΠϧΛ࢖͏Α͏ͳ࢓૊Έʹ͢ΔͨΊ
  24. ؊ESZSVO͕௨͔ͬͨΒͱ͍ͬͯʜ • ࣮ߦ͕੒ޭ͢Δͱ͸ݶΒͳ͍ • ͜ͷ৔߹͸མͪண͍ͯΤϥʔ಺༰ΛݟΔ • τϨλͰ͸AWSϦιʔεͷ্ݶʹୡͨ͠৔߹ͳͲʹ௚໘ ͨ͠(EC2Πϯελϯε਺ͷ্ݶ…etc)

  25. ؊͢΂ͯΛUFSSBGPSN؅ཧԼʹ͸ஔ͔ͳ͍ • tfstateϑΝΠϧΛਖ਼ͱͯ͠ಈ͘ͷͰঢ়ଶ͕มԽ͢ΔASG ͳͲʹ͸ෆ޲͖(ͩͱࢥ͍ͬͯ·͢) • ͦΕͱ…ͩͬͯා͍Μͩ΋Μ

  26. ࣮ࡍੲ͸݁ߏා͔ͬͨ • ͔ͭͯ͸༧ظͤ͵ഁյతͳڍಈ͕͋ͬͨ

  27. 3PBEXPSLFS

  28. 3PBEXPSLFS • https://github.com/winebarrel/ • Route53ͷϨίʔυઃఆΛRubyͷDSLͰهड़Ͱ͖Δ

  29. 3PBEXPSLFS • Engineyard࣌୅͸EngineyardͱAWSͷϋΠϒϦουӡ༻ • Engineyard: EC2Πϯελϯε(App, Redis, MySQL…) • AWS:

    Route53, S3… • ೖࣾ࣌ʹAWS্ͷ΋ͷͷίʔυԽʹணखͨ͠ • ࠷ॳ͔ΒTerraformʹ͠ͳ͔ͬͨͷ͸౰࣌ͷTerraform͸ طଘͷDNSϨίʔυͷexport͕Ͱ͖ͳ͔ͬͨ • Roadworker͸Ͱ͖ͨ
  30. 1BDLFS

  31. 1BDLFS • https://www.packer.io/ • Hashicorp੡ϓϩμΫτ • AMI(AWS), Virtualbox, dockerͳͲ֤छΠϝʔδΛϓϩά ϥϚϒϧʹ࡞੒Ͱ͖Δ

    • jsonͰهड़
  32. 1BDLFS • τϨλͰͷ༻్ • ϕʔεͱͳΔAMIͷ࡞੒ • ։ൃ؀ڥ༻Vagrant boxͷ࡞੒ • Packerʹ͍ͭͯ͸͜Εʹ୅ΘΔ΋ͷ͸ແ͍(͸ͣ)

  33. 1BDLFSͰ΍͍ͬͯΔ͜ͱ • OSͷॳظઃఆ • RubyͷΠϯετʔϧ • ϛυϧ΢ΣΞͷΠϯετʔϧ΍ίϯϑΟάͷ഑෍͸ Ansibleʹ΍Β͍ͤͯΔ • Ҏલ͸ϛυϧ΢ΣΞͷΠϯετʔϧ΋Packerʹ΍Β

    ͤͯશ෦ೖΓͷAMIΛ࡞͍ͬͯͨ(ϒϩάهࣄΑΓ↓)
  34. શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ • શ෦ೖΓʹͨ͠৔߹ͷ… • ϝϦοτ • ىಈ͢Δ͚ͩͰαʔϏε౤ೖՄೳͱͳΔ • σϝϦοτ •

    ίϯϑΟάมߋͷͨͼʹAMIΛࣽࠐΉඞཁ͕͋Δ • ·ͨͦͷͨͼʹΠϯελϯεશೖΕସ͑ • ͜ͷ࢓૊ΈΛࣗಈԽ͢Ε͹σϝϦοτ͸ͳ͘ͳΔ͔΋͠Εͳ͍ • ͨͩࣽ͜͠Ή࣌ؒΛ଴ͭͷ͕μϧ͍ͱ͖΋͋Δ…
  35. શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ • શ෦ೖΓͷํ͕Ϋϥ΢υతͰ͸͋Δ • ίϯςφͷӡ༻ͱۙ͘ͳΔ͸ͣ • ͭ·Γঢ়ଶͷมߋ = ΠϯελϯεΛ৽͘͠࡞ΔˍೖΕସ͑Δ •

    ίϯϑΟάྨͷมߋΛAnsible/chefͳͲͰద༻Λ܁Γฦ͢ͷ͸ݩ ͸ΦϯϓϨͷߟ͑ํ • ͭ·Γಉ͡αʔόΛ࢖͍ճ͢ͱ͍͏લఏͷιϦϡʔγϣϯ • ͍ΘΏΔႈ౳ੑ • Ϋϥ΢υ͸”ࣺͯΔ”ͱ͍͏બ୒ࢶ͕͋Δ • ࣺͯΔ->࠶ߏங͕୹࣌ؒͰՄೳ • ΋ͪΖΜΠϯελϯε͕յΕΔ·Ͱ࢖͍ճ͢͜ͱ΋Ͱ͖Δ
  36. "OTJCMF

  37. "OTJCMF • https://www.ansible.com/ • αʔόߏஙͷࣗಈԽɺႈ౳ੑͷ୲อ • ಉ౳ͷπʔϧʹchef΍itamae΍puppet͕͋Δ • yamlͰهड़

  38. ͳͥ"OTJCMF͔ʁ • ผʹͳΜͰ΋ྑ͔ͬͨ • Α͋͘Δ(?)ࣄҊ • αʔόߏஙεΫϦϓτ͕ൿ఻ͷλϨͩᵆ( :^o^)ᵒ • αʔόߏஙखॱॻ͕ոจॻͩᵎ(^o^;

    )ᵊ • chef/ansibleΛಋೖͩ(^q^) • cookbook/playbook͕ݹจॻʹͳͬͨʗ(^o^)ʘ • ͜ͷखͷπʔϧ͸ͲΕ࢖ͬͯ΋هड़ϧʔϧ΍ӡ༻ϧʔϧΛܾΊ͓͔ͯͳ͍ ͱυπϘʹϋϚΔ(ܦݧ্) • Πϯϑϥίʔυ΋ͪΌΜͱϨϏϡʔ͢Δʢ͋ͨΓ·͑Ͱ͕͢…ʣ • ςετɺServerspecΛॻ͍͓ͯ͜͏ • ʮ༨ܭͳ͜ͱ͕Ͱ͖ͳ͍puppet͕Ұ൪ྑ͍ʯbyಉۀऀͷ༑ਓ ͱ͍͏ҙݟ΋ ͋Δ
  39. "OTJCMF • τϨλͰ͸… • EC2ʹରͯ͠ϛυϧ΢ΣΞͷΠϯετʔϧɺίϯϑΟάͷ ഑෍Λߦ͍ͬͯΔ • μΠφϛοΫΠϯϕϯτϦΛར༻ • EC2ʹ෇༩ͨ͠ServiceλάͱRoleλάͰϓϩϏδϣχ

    ϯάର৅ͷϗετͱϩʔϧΛಈతʹऔಘ • ϗετͷ૿ݮɺIPͷมԽΛҙࣝͤͣʹϓϩϏδϣχϯ άͰ͖Δ
  40. 4FSWFSTQFD

  41. 4FSWFSTQFD • http://serverspec.org/ • ίϯϑΟάͷςετϑϨʔϜϫʔΫ • ϦϑΝΫλϦϯά͕ओ໨తͷιϑτ΢ΣΞ • RSpecͰهड़

  42. 4FSWFSTQFD • τϨλͰ΋ΠϯϑϥίʔυΛϦϑΝΫλϦϯά͠΍ ͘͢͢ΔͨΊʹॻ͍͍ͯΔ • Serverspec͕ॻ͍ͯ͋Ε͹ྫ͑͹ansible->chef΁ͷ ৐Γ׵͑(ͨͿΜ΍Βͳ͍͚Ͳ)΋΍Γ΍͍͢

  43. "VSPSB

  44. "VSPSB • RDSͷΤϯδϯͷҰͭ • MySQLޓ׵ • GUIϙνϙνͰߏஙͯ͠·͢ • ↓ͱ͍͏ߟ͑(ݴ͍༁ʁ)

  45. "VSPSB • ಋೖʹ͋ͨͬͯݕূͨ͜͠ͱ • Ҡߦલ(MySQL on Engineyard)ͱ࣮ߦܭը͕มΘΒͳ͍͜ͱͷ֬ೝ • ͪΌΜͱࠓ·Ͱ௨ΓʹΠϯσοΫεΛ࢖ͬͯ͘ΕΔ͔ •

    ίϯϑΟάϨʔγϣϯͷਫ਼ࠪ • MySQL͓͡͞Μͱͯ͠͸ඇৗʹؾʹͳͬͨ… • Ͳ͜ΛͲ͏มߋ͔ͨ͠͸ϒϩάهࣄΛࢀর͍ͯͩ͘͠͞ • εέʔϧΞοϓɺσʔλϦΧόϦ…ͳͲͷΦϖखॱɺཁ͢Δ࣌ؒͳͲ • ༗ࣄʹඋ͑ͯखॱΛཱ͓֬ͯͨ͘͠Ί • ཁ͸ૉৼΓɻ࿅शɻ
  46. "VSPSB • ੑೳࢼݧ͸΍ͬͯͳ͍ɻͱ͍͏͔΍Βͳ͍ɻ • sysbenchͳͲͷࢼݧʹ͍ͭͯ͸͢Ͱʹଟ͘ͷهࣄ͕ωο τʹ্͕͍ͬͯΔɻਓͱಉ͜͡ͱΛͯ͠΋ҙຯ͕ͳ͍ɻ • ΋͠΍ΔͳΒຊ൪૬౰ͷσʔλྔɺSQLɺτϥϑΟοΫ • όονॲཧͷࢀরܥΛAuroraʹ޲͚ͯෛՙঢ়گΛ֬ೝ

    • ΦϯϥΠϯॲཧʹ͍ͭͯ͸લεϥΠυͷ௨Γ࣮ߦܭըͷ ֬ೝΛߦͬͯMySQLͱಉ౳Ͱ͋Δ͜ͱΛ֬ೝͯ͠OKͱ͠ ͨ
  47. "VSPSBͰ͸࣮ݱͰ͖ͳ͍͜ͱ • θϩμ΢ϯλΠϜ • ϝϯςφϯε΢Οϯυ΢͕͋ΔͷͰ͍͔ͭ͸࠶ىಈ͠ͳ͖Ό ͳΒͳ͍ͱ͖͕དྷΔɻ • Ͱ΋ϑΣΠϧΦʔόͰ1෼ఔ౓Ͱ׬ • 1෼΋ࢭΊͨ͘ͳ͍৔߹͸…

    • MySQL on EC2 with MHA • ࢲͷهԱͰ͸MHAͷํ͕ϑΣΠϧΦʔό͸ૣ͍ • ׬શθϩμ΢ϯλΠϜΛ໨ࢦ͔ͨͬͨ͠ΒϚϧνϚελͳ෼ ࢄDBͰ΋…Cassandraͱ͔Ͱ…ؤுͬͯ…͍ͩ͘͞…
  48. $JSDMF$*

  49. $JSDMF$* • ஶ໊ͳCIαʔϏεͷҰͭ • RailsΞϓϦέʔγϣϯͷCI/σϓϩΠج൫ͱͯ͠׆༻ • ΠϯϑϥΦϖϨʔγϣϯͷ࣮ߦج൫ͱͯ͠΋׆༻

  50. $JSDMF$*͔ΒͷσϓϩΠ • EC2ʹ෇༩͞ΕͨServiceλάͱRoleλάͰର৅ͷϗ ετΛಛఆͯ͠Capistrano΍AnsibleΛ࣮ߦ • ·͋ී௨ͷ΍ΓํͰ͢ • θϩμ΢ϯλΠϜσϓϩΠͷ࣮ݱ͸ʁ • unicorn͸ϗοτσϓϩΠ͕ޮ͘

    • ࠓͷن໛Ͱ͸͜ΕͰे෼
  51. ࠓޙ

  52. ࠓޙ • ίϯςφʁ • ΍ΔͳΒ৽نαʔϏε։ൃ࣌ • ։ൃख๏ɺσϓϩΠํ๏ɺӡ༻ख๏͕มԽ͢ΔɺଞΤϯδχΞʹ΋ͦΕΛ΍ͬ ͯ΋Β͏ඞཁ͕͋Δ • طଘͷαʔϏεΛίϯςφԽ͢Δ͜ͱ͸ࠓ͸ߟ͍͑ͯͳ͍

    • ৽نαʔϏεͰίϯςφಋೖͷޮՌ͕ग़ͨΒ΍Δ͔΋ • ASGʁ • ୆਺͕૿͑ͯඇϐʔΫ࣌ͷίετ࡟ݮޮՌ͕ݟ͑ͦ͏ʹͳͬͨΒ • ͨͩ͠Terraformͱͷ૬ੑ͕ѱͦ͏ͩ
  53. ͓ΘΓ