$30 off During Our Annual Pro Sale. View Details »

トレタのインフラ運用

 トレタのインフラ運用

【トレタ×プレイド】TechBlog Deep Dive Meetup #1 〜インフラ編〜
http://plaidtech.connpass.com/event/33511/

での発表内容。
Terraform, Packer, Roadworker, Ansible, Serverspec, Circle CIなどをトレタでどう使っているか?何が罠か?など。

Hiroaki Sano

July 06, 2016
Tweet

More Decks by Hiroaki Sano

Other Decks in Technology

Transcript

  1. τϨλͷΠϯϑϥӡ༻
    ʲτϨλºϓϨΠυʳ5FDI#MPH%FFQ%JWF.FFUVQ

    )JSPBLJ4BOP

    View Slide

  2. NF
    • Name:
    • ࠤ໺༟ষ(Hiroaki Sano)
    • Personal website:
    • https://hiroakis.com/blog/
    • Company:
    • NEC Soft, Ltd. (2006/4-)
    • CyberAgent, Inc. (2011/3-)
    • Toreta, Inc. (2014/11-)

    View Slide

  3. View Slide

  4. View Slide

  5. τϨλ
    • ҿ৯ళ޲͚༧໿؅ཧΞϓϦέʔγϣϯΛiPadΞϓϦͱͯ͠ఏڙ
    • ҰൠϢʔβ޲͚ʹ΢Σϒ༧໿΋ఏڙ
    • ސ٬ɿϨετϥϯɺډञ԰ͳͲͷҿ৯ళ
    • ྫ͑͹…
    • Զͷגࣜձ༷ࣾ(ԶͷΠλϦΞϯɺԶͷϑϨϯν…etc)
    • גࣜձࣾେঙ༷(ঙ΍ɺ΍Δ͖஡԰…etc)
    • ΤʔɾϐʔΧϯύχʔ༷(௩ా೶৔…etc)
    • ϩʔετϗʔε༷
    • etc
    • ϩʔϯν͸2013೥12݄
    • ಋೖళฮ਺ɿ6000ళฮʙ
    • ళฮ͋ͨΓͷֹ݄՝ۚ
    • ίʔϙϨʔτΧϥʔ͸྘
    • ΦϑΟε͸ौ୩ -> ໨ࠇ -> ܙൺण -> ޒ൓ా

    View Slide

  6. ͜ΕΛ
    CONFIDENTIAL - Toreta Inc., All Right Reserved.
    Toreta Inc., All Right Reserved.

    View Slide

  7. ͜͏
    CONFIDENTIAL - Toreta Inc., All Right Reserved.
    Toreta Inc., All Right Reserved.

    View Slide

  8. τϨλͷίϯηϓτ
    • ҿ৯ళͷ༧໿؅ཧɺސ٬؅ཧΛITԽ
    • ҿ৯ళͰಇ͘ਓͷۀ຿ͷࣗಈԽ
    • ༧໿ࣄނΛ๷͙

    View Slide

  9. ࠓ೔͸ٕज़ϒϩάਂ۷Γձ
    ͱ͍͏͜ͱͰʜ

    View Slide

  10. τϨλ։ൃऀϒϩά
    IUUQUFDIUPSFUBJO

    View Slide

  11. View Slide

  12. ࢲͷॻ͍ͨهࣄ
    • Raspberry PiͱԹ౓ɾ࣪౓ɾޫηϯαʔͰΦϑΟεͷঢ়گΛՄࢹԽ͢Δ
    • http://tech.toreta.in/entry/2016/02/15/153203
    • τϨλͷΠϯϑϥӡ༻ɺࢧ͍͑ͯΔಓ۩(Packer, Terraform,
    Serverspec, Ansible, Roadworker, Circle CI)ɺߟ͑ํ
    • http://tech.toreta.in/entry/2016/04/14/143248
    • ϝʔϧͷ഑৴ঢ়گΛՄࢹԽɺ௥੻͢Δ
    • http://tech.toreta.in/entry/2016/04/20/153852
    • Engineyard͔ΒAWSʹҠઃͯ͠Auroraͷӡ༻Λ։࢝ͨ͠
    • http://tech.toreta.in/entry/2016/06/16/114919

    View Slide

  13. ࠓ೔࿩͢͜ͱ
    • ओʹAWSͰͷΠϯϑϥӡ༻ʹ͍ͭͯ
    • τϨλͷAPIαʔόͷΤίγεςϜ
    • ओʹϒϩάωλʹొ৔ͨ͠ࣄฑΛத৺ʹ…
    • Terraform
    • Roadworker
    • Packer
    • Ansible
    • Serverspec
    • Aurora
    • Circle CI
    • ࠓޙʹ͍ͭͯ
    • ٕज़తʹਅ৽͍͠΋ͷ͸ͳ͍Ͱ͕͢ࢀߟʹ͍͚ͯͨͩ͠Ε͹…

    View Slide

  14. τϨλ"1*ͱͦͷपลͷΤίγεςϜ

    View Slide

  15. Amazon
    Route 53
    Amazon
    RDS(Aurora)
    api

    worker
    Redis
    memcached

    Amazon
    SNS
    Amazon SQS
    τϨλ"1*ͱͦͷपลͷΤίγεςϜ
    FAX/SMS…
    Amazon
    S3
    ը૾/Ի੠…
    ଞαϒγεςϜ΁ͷ௨஌
    ϩά
    ϝʔϧ
    fax, sms
    API
    worker
    fluentd

    View Slide

  16. τϨλ"1*ͱͦͷपลͷΤίγεςϜ
    • Amazon Web Services
    • 5݄ʹEngineyard͔ΒҠઃ(ϒϩάهࣄͷ௨Γ)
    • Appαʔό: APIɺWorkerಉډ. nginx, Rails(unicorn), sidekiqͰ࣮૷.
    • Redis: WorkerͷΩϡʔΠϯά༻
    • memcached: Ωϟογϡ
    • RDS Aurora: ϝΠϯͷσʔλϕʔε
    • Google Cloud Platform
    • BigQuery: fluentd͔Βϩάͷసૹ
    • ιʔϦʔαʔό΍Ұ෦ͷόοναʔό: GCEͷݕূΛ݉Ͷͯͪ͜Βʹߏஙͯ͋͠Δ
    • Monitoring
    • Mackerel: Ϧιʔε؂ࢹɺϝτϦΫε
    • Pingdom: ΤϯυϙΠϯτ؂ࢹ
    • Pagerduty: ΞϥʔτରԠͷεέδϡʔϦϯάɺి࿩௨஌
    • CI
    • Circle CI: CI͓ΑͼσϓϩΠɺ͓ΑͼΦϖϨʔγϣϯͷத৺

    View Slide

  17. app

    worker
    Redis
    Aurora
    Writer
    memcached

    app

    worker
    Redis memcached

    Availability Zone A
    71$ઃܭ
    public subnet
    Availability Zone C
    Aurora
    Reader
    public subnet
    10.0.0.0/16
    10.0.0.0/24 10.0.1.0/24

    View Slide

  18. 71$ઃܭ
    • ֤ίϯϙʔωϯτ͸ϚϧνAZͰ഑ஔ
    • 10.0.X.0/24 -> AZ-A
    • 10.0.Y.0/24 -> AZ-C
    • ύϒϦοΫαϒωοτͷΈ
    • RDSͳͲ΋ύϒϦοΫαϒωοτʹஔ͘
    • ݎ࿚ੑ͸ηΩϡϦςΟάϧʔϓͰ୲อ
    • ࡶ͡Όͳ͍͔ʁͱͨ·ʹݴΘΕΔ
    • ݱঢ়ͷ࡞ΓͰผʹࠔ͍ͬͯͳ͍
    • ࠔͬͨΒ࡞Γͳ͓͢

    View Slide

  19. 5FSSBGPSN

    View Slide

  20. 5FSSBGPSN
    • https://www.terraform.io/
    • Hashicorp੡ϓϩμΫτ
    • AWS΍GCPͳͲ֤छΫϥ΢υͷঢ়ଶΛίʔυͰهड़͢Δ΋ͷ
    • HCLͱ͍͏jsonϥΠΫͳϑΥʔϚοτͰهड़
    ग़య: https://www.terraform.io/

    View Slide

  21. 5FSSBGPSN
    • VPC, SecurityGroup, EC2Λ؅ཧର৅ͱ͍ͯ͠Δ
    • EC2ߏங࣌͸Environment, Role, ServiceλάΛ෇༩͢Δ
    • ͜ΕΒ͸σϓϩΠ΍ϓϩϏδϣχϯάͰ࢖͏
    • AWSͷ࢖͍ํͱͯ͠యܕతͱ͍͏͔ݹయతͳखஈ

    View Slide

  22. ͳͥ5FSSBGPSN͔ʁ
    • CloudFormationΑΓ͸׳Ε͍͔ͯͨΒ
    • Google Cloud Platform΋࢖͍͔ͬͯͨΒ
    • ͪͳΈʹࣅͨΑ͏ͳπʔϧʹ͜Μͳͷ΋͋Γ·͢
    • Apache Libcloud
    • https://libcloud.apache.org/index.html
    • छʑͷΫϥ΢υʹରԠͨ͠boto(aws sdk for Python)ͷ
    Α͏ͳ΋ͷ

    View Slide

  23. ؊UGTUBUFϑΝΠϧͷѻ͍
    • terraform࣮ߦޙͷঢ়ଶ͕هड़͞Ε͍ͯΔϑΝΠϧ
    • terraform͸͜ͷϑΝΠϧͷঢ়ଶΛਖ਼ͱͯ͠มߋΛద༻͢Δ
    • terraform࣮ߦ࣌ʹ͜ͷϑΝΠϧ͕յΕͨΓઌ૆ฦΓͨ͠ঢ়ଶͰ࣮ߦ͢Δͱ
    ࣄނΔ
    • ͜ͷϑΝΠϧ͕յΕͨΒʁר͖໭ͬͯ͠·ͬͨΒʁ => ؤு࣏ͬͯͦ
    ͏:(ʀƄƅ’Тƅ'):
    • खݩ͔Β࣮ߦ͢Δ৔߹͸githubͳͲͰ؅ཧͭͭ͠࡞ۀऀ֤Ґ͕ඞͣ࠷৽ͷ
    ΋ͷΛpull͔ͯ͠Β࢖͏
    • τϨλͰ͸…
    • S3ʹஔ͖ͭͭɺCircle CI͔ΒterraformΛ࣮ߦ
    • ΦϖϨʔγϣϯΛ࣮ߦ͢Δਓ෺͸Ұਓ(Circle CI)ʹ͢ΔͨΊ
    • ඞͣ࠷৽ͷtfstateϑΝΠϧΛ࢖͏Α͏ͳ࢓૊Έʹ͢ΔͨΊ

    View Slide

  24. ؊ESZSVO͕௨͔ͬͨΒͱ͍ͬͯʜ
    • ࣮ߦ͕੒ޭ͢Δͱ͸ݶΒͳ͍
    • ͜ͷ৔߹͸མͪண͍ͯΤϥʔ಺༰ΛݟΔ
    • τϨλͰ͸AWSϦιʔεͷ্ݶʹୡͨ͠৔߹ͳͲʹ௚໘
    ͨ͠(EC2Πϯελϯε਺ͷ্ݶ…etc)

    View Slide

  25. ؊͢΂ͯΛUFSSBGPSN؅ཧԼʹ͸ஔ͔ͳ͍
    • tfstateϑΝΠϧΛਖ਼ͱͯ͠ಈ͘ͷͰঢ়ଶ͕มԽ͢ΔASG
    ͳͲʹ͸ෆ޲͖(ͩͱࢥ͍ͬͯ·͢)
    • ͦΕͱ…ͩͬͯා͍Μͩ΋Μ

    View Slide

  26. ࣮ࡍੲ͸݁ߏා͔ͬͨ
    • ͔ͭͯ͸༧ظͤ͵ഁյతͳڍಈ͕͋ͬͨ

    View Slide

  27. 3PBEXPSLFS

    View Slide

  28. 3PBEXPSLFS
    • https://github.com/winebarrel/
    • Route53ͷϨίʔυઃఆΛRubyͷDSLͰهड़Ͱ͖Δ

    View Slide

  29. 3PBEXPSLFS
    • Engineyard࣌୅͸EngineyardͱAWSͷϋΠϒϦουӡ༻
    • Engineyard: EC2Πϯελϯε(App, Redis, MySQL…)
    • AWS: Route53, S3…
    • ೖࣾ࣌ʹAWS্ͷ΋ͷͷίʔυԽʹணखͨ͠
    • ࠷ॳ͔ΒTerraformʹ͠ͳ͔ͬͨͷ͸౰࣌ͷTerraform͸
    طଘͷDNSϨίʔυͷexport͕Ͱ͖ͳ͔ͬͨ
    • Roadworker͸Ͱ͖ͨ

    View Slide

  30. 1BDLFS

    View Slide

  31. 1BDLFS
    • https://www.packer.io/
    • Hashicorp੡ϓϩμΫτ
    • AMI(AWS), Virtualbox, dockerͳͲ֤छΠϝʔδΛϓϩά
    ϥϚϒϧʹ࡞੒Ͱ͖Δ
    • jsonͰهड़

    View Slide

  32. 1BDLFS
    • τϨλͰͷ༻్
    • ϕʔεͱͳΔAMIͷ࡞੒
    • ։ൃ؀ڥ༻Vagrant boxͷ࡞੒
    • Packerʹ͍ͭͯ͸͜Εʹ୅ΘΔ΋ͷ͸ແ͍(͸ͣ)

    View Slide

  33. 1BDLFSͰ΍͍ͬͯΔ͜ͱ
    • OSͷॳظઃఆ
    • RubyͷΠϯετʔϧ
    • ϛυϧ΢ΣΞͷΠϯετʔϧ΍ίϯϑΟάͷ഑෍͸
    Ansibleʹ΍Β͍ͤͯΔ
    • Ҏલ͸ϛυϧ΢ΣΞͷΠϯετʔϧ΋Packerʹ΍Β
    ͤͯશ෦ೖΓͷAMIΛ࡞͍ͬͯͨ(ϒϩάهࣄΑΓ↓)

    View Slide

  34. શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ
    • શ෦ೖΓʹͨ͠৔߹ͷ…
    • ϝϦοτ
    • ىಈ͢Δ͚ͩͰαʔϏε౤ೖՄೳͱͳΔ
    • σϝϦοτ
    • ίϯϑΟάมߋͷͨͼʹAMIΛࣽࠐΉඞཁ͕͋Δ
    • ·ͨͦͷͨͼʹΠϯελϯεશೖΕସ͑
    • ͜ͷ࢓૊ΈΛࣗಈԽ͢Ε͹σϝϦοτ͸ͳ͘ͳΔ͔΋͠Εͳ͍
    • ͨͩࣽ͜͠Ή࣌ؒΛ଴ͭͷ͕μϧ͍ͱ͖΋͋Δ…

    View Slide

  35. શ෦ೖΓͷ".*ʹ͢Δ΂͖͔ʁ
    • શ෦ೖΓͷํ͕Ϋϥ΢υతͰ͸͋Δ
    • ίϯςφͷӡ༻ͱۙ͘ͳΔ͸ͣ
    • ͭ·Γঢ়ଶͷมߋ = ΠϯελϯεΛ৽͘͠࡞ΔˍೖΕସ͑Δ
    • ίϯϑΟάྨͷมߋΛAnsible/chefͳͲͰద༻Λ܁Γฦ͢ͷ͸ݩ
    ͸ΦϯϓϨͷߟ͑ํ
    • ͭ·Γಉ͡αʔόΛ࢖͍ճ͢ͱ͍͏લఏͷιϦϡʔγϣϯ
    • ͍ΘΏΔႈ౳ੑ
    • Ϋϥ΢υ͸”ࣺͯΔ”ͱ͍͏બ୒ࢶ͕͋Δ
    • ࣺͯΔ->࠶ߏங͕୹࣌ؒͰՄೳ
    • ΋ͪΖΜΠϯελϯε͕յΕΔ·Ͱ࢖͍ճ͢͜ͱ΋Ͱ͖Δ

    View Slide

  36. "OTJCMF

    View Slide

  37. "OTJCMF
    • https://www.ansible.com/
    • αʔόߏஙͷࣗಈԽɺႈ౳ੑͷ୲อ
    • ಉ౳ͷπʔϧʹchef΍itamae΍puppet͕͋Δ
    • yamlͰهड़

    View Slide

  38. ͳͥ"OTJCMF͔ʁ
    • ผʹͳΜͰ΋ྑ͔ͬͨ
    • Α͋͘Δ(?)ࣄҊ
    • αʔόߏஙεΫϦϓτ͕ൿ఻ͷλϨͩᵆ( :^o^)ᵒ
    • αʔόߏஙखॱॻ͕ոจॻͩᵎ(^o^; )ᵊ
    • chef/ansibleΛಋೖͩ(^q^)
    • cookbook/playbook͕ݹจॻʹͳͬͨʗ(^o^)ʘ
    • ͜ͷखͷπʔϧ͸ͲΕ࢖ͬͯ΋هड़ϧʔϧ΍ӡ༻ϧʔϧΛܾΊ͓͔ͯͳ͍
    ͱυπϘʹϋϚΔ(ܦݧ্)
    • Πϯϑϥίʔυ΋ͪΌΜͱϨϏϡʔ͢Δʢ͋ͨΓ·͑Ͱ͕͢…ʣ
    • ςετɺServerspecΛॻ͍͓ͯ͜͏
    • ʮ༨ܭͳ͜ͱ͕Ͱ͖ͳ͍puppet͕Ұ൪ྑ͍ʯbyಉۀऀͷ༑ਓ ͱ͍͏ҙݟ΋
    ͋Δ

    View Slide

  39. "OTJCMF
    • τϨλͰ͸…
    • EC2ʹରͯ͠ϛυϧ΢ΣΞͷΠϯετʔϧɺίϯϑΟάͷ
    ഑෍Λߦ͍ͬͯΔ
    • μΠφϛοΫΠϯϕϯτϦΛར༻
    • EC2ʹ෇༩ͨ͠ServiceλάͱRoleλάͰϓϩϏδϣχ
    ϯάର৅ͷϗετͱϩʔϧΛಈతʹऔಘ
    • ϗετͷ૿ݮɺIPͷมԽΛҙࣝͤͣʹϓϩϏδϣχϯ
    άͰ͖Δ

    View Slide

  40. 4FSWFSTQFD

    View Slide

  41. 4FSWFSTQFD
    • http://serverspec.org/
    • ίϯϑΟάͷςετϑϨʔϜϫʔΫ
    • ϦϑΝΫλϦϯά͕ओ໨తͷιϑτ΢ΣΞ
    • RSpecͰهड़

    View Slide

  42. 4FSWFSTQFD
    • τϨλͰ΋ΠϯϑϥίʔυΛϦϑΝΫλϦϯά͠΍
    ͘͢͢ΔͨΊʹॻ͍͍ͯΔ
    • Serverspec͕ॻ͍ͯ͋Ε͹ྫ͑͹ansible->chef΁ͷ
    ৐Γ׵͑(ͨͿΜ΍Βͳ͍͚Ͳ)΋΍Γ΍͍͢

    View Slide

  43. "VSPSB

    View Slide

  44. "VSPSB
    • RDSͷΤϯδϯͷҰͭ
    • MySQLޓ׵
    • GUIϙνϙνͰߏஙͯ͠·͢
    • ↓ͱ͍͏ߟ͑(ݴ͍༁ʁ)

    View Slide

  45. "VSPSB
    • ಋೖʹ͋ͨͬͯݕূͨ͜͠ͱ
    • Ҡߦલ(MySQL on Engineyard)ͱ࣮ߦܭը͕มΘΒͳ͍͜ͱͷ֬ೝ
    • ͪΌΜͱࠓ·Ͱ௨ΓʹΠϯσοΫεΛ࢖ͬͯ͘ΕΔ͔
    • ίϯϑΟάϨʔγϣϯͷਫ਼ࠪ
    • MySQL͓͡͞Μͱͯ͠͸ඇৗʹؾʹͳͬͨ…
    • Ͳ͜ΛͲ͏มߋ͔ͨ͠͸ϒϩάهࣄΛࢀর͍ͯͩ͘͠͞
    • εέʔϧΞοϓɺσʔλϦΧόϦ…ͳͲͷΦϖखॱɺཁ͢Δ࣌ؒͳͲ
    • ༗ࣄʹඋ͑ͯखॱΛཱ͓֬ͯͨ͘͠Ί
    • ཁ͸ૉৼΓɻ࿅शɻ

    View Slide

  46. "VSPSB
    • ੑೳࢼݧ͸΍ͬͯͳ͍ɻͱ͍͏͔΍Βͳ͍ɻ
    • sysbenchͳͲͷࢼݧʹ͍ͭͯ͸͢Ͱʹଟ͘ͷهࣄ͕ωο
    τʹ্͕͍ͬͯΔɻਓͱಉ͜͡ͱΛͯ͠΋ҙຯ͕ͳ͍ɻ
    • ΋͠΍ΔͳΒຊ൪૬౰ͷσʔλྔɺSQLɺτϥϑΟοΫ
    • όονॲཧͷࢀরܥΛAuroraʹ޲͚ͯෛՙঢ়گΛ֬ೝ
    • ΦϯϥΠϯॲཧʹ͍ͭͯ͸લεϥΠυͷ௨Γ࣮ߦܭըͷ
    ֬ೝΛߦͬͯMySQLͱಉ౳Ͱ͋Δ͜ͱΛ֬ೝͯ͠OKͱ͠
    ͨ

    View Slide

  47. "VSPSBͰ͸࣮ݱͰ͖ͳ͍͜ͱ
    • θϩμ΢ϯλΠϜ
    • ϝϯςφϯε΢Οϯυ΢͕͋ΔͷͰ͍͔ͭ͸࠶ىಈ͠ͳ͖Ό
    ͳΒͳ͍ͱ͖͕དྷΔɻ
    • Ͱ΋ϑΣΠϧΦʔόͰ1෼ఔ౓Ͱ׬
    • 1෼΋ࢭΊͨ͘ͳ͍৔߹͸…
    • MySQL on EC2 with MHA
    • ࢲͷهԱͰ͸MHAͷํ͕ϑΣΠϧΦʔό͸ૣ͍
    • ׬શθϩμ΢ϯλΠϜΛ໨ࢦ͔ͨͬͨ͠ΒϚϧνϚελͳ෼
    ࢄDBͰ΋…Cassandraͱ͔Ͱ…ؤுͬͯ…͍ͩ͘͞…

    View Slide

  48. $JSDMF$*

    View Slide

  49. $JSDMF$*
    • ஶ໊ͳCIαʔϏεͷҰͭ
    • RailsΞϓϦέʔγϣϯͷCI/σϓϩΠج൫ͱͯ͠׆༻
    • ΠϯϑϥΦϖϨʔγϣϯͷ࣮ߦج൫ͱͯ͠΋׆༻

    View Slide

  50. $JSDMF$*͔ΒͷσϓϩΠ
    • EC2ʹ෇༩͞ΕͨServiceλάͱRoleλάͰର৅ͷϗ
    ετΛಛఆͯ͠Capistrano΍AnsibleΛ࣮ߦ
    • ·͋ී௨ͷ΍ΓํͰ͢
    • θϩμ΢ϯλΠϜσϓϩΠͷ࣮ݱ͸ʁ
    • unicorn͸ϗοτσϓϩΠ͕ޮ͘
    • ࠓͷن໛Ͱ͸͜ΕͰे෼

    View Slide

  51. ࠓޙ

    View Slide

  52. ࠓޙ
    • ίϯςφʁ
    • ΍ΔͳΒ৽نαʔϏε։ൃ࣌
    • ։ൃख๏ɺσϓϩΠํ๏ɺӡ༻ख๏͕มԽ͢ΔɺଞΤϯδχΞʹ΋ͦΕΛ΍ͬ
    ͯ΋Β͏ඞཁ͕͋Δ
    • طଘͷαʔϏεΛίϯςφԽ͢Δ͜ͱ͸ࠓ͸ߟ͍͑ͯͳ͍
    • ৽نαʔϏεͰίϯςφಋೖͷޮՌ͕ग़ͨΒ΍Δ͔΋
    • ASGʁ
    • ୆਺͕૿͑ͯඇϐʔΫ࣌ͷίετ࡟ݮޮՌ͕ݟ͑ͦ͏ʹͳͬͨΒ
    • ͨͩ͠Terraformͱͷ૬ੑ͕ѱͦ͏ͩ

    View Slide

  53. ͓ΘΓ

    View Slide