SRE的team開発Tipsとベストプラクティスっぽい何か

1333e8256b0ba80c369f4bcc98a839d7?s=47 k-oguma
January 18, 2019

 SRE的team開発Tipsとベストプラクティスっぽい何か

SRE Lounge #7の発表資料です。

1333e8256b0ba80c369f4bcc98a839d7?s=128

k-oguma

January 18, 2019
Tweet

Transcript

  1. 2.

    ϝϧΧϦSREͷ୲౰ൣғ 2 ࠓ೔ͷτϐοΫ 01 SREsΛߟ͑Δ 02 Toilͷଊ͑ํ (ݾΛ஌Δ) 03 ઓུతSRE૊৫࿦

    04 SREۀ຿Ͱ΋औΓೖΕͨํ͕ྑ͍จԽ 05 GitΛ࢖ͬͨ։ൃͱCodeReview࠶ߟ 06 ·ͱΊ 07
  2. 4.

    history 4 history % 1 Work as a system engineer

    and PMO assistant 2 Infra engineer at DC enterprise 3 Infra engineer at Game development 4 SRE at Mercari, Inc.
  3. 11.

    ϝϧΧϦSREͷ୲౰ൣғ 11 • Monolithic architectureͷߏங΍ӡ༻อक(Devops)ɺ৴པੑͷ޲্ • ର৅Region = JP /

    US / UK • ͘͞ΒΠϯλʔωοτ, AWS, GCP • Microservices ʹ͍ͭͯ͸ɺޙ௥͍ͷ։࢝଴ͪɺΩϟονΞοϓظؒ • ࠓޙɺ֤υϝΠϯνʔϜʹ഑ଐ༧ఆɻৄࡉ͸ޙ΄Ͳɻ • Data platform։ൃɺDevOps of Data (ETL) • Security ରԠ • Middleware, Tool։ൃ
  4. 12.

    ϝϧΧϦSREͷ୲౰ൣғ 12 • Middleware, Tool։ൃ • Productͷػೳ։ൃ • e.g. cache

    control, DBIपΓͷߋ৽, maintenance mode࡞੒, etc • BI systemӡ༻ • ։ൃऀ͔ΒͷReviewґཔͷରԠͳͲ • Πϯϑϥઃܭ • Search Ops • ML Ops • On-Call • etc
  5. 14.

    ϝϧΧϦSREͷ୲౰ൣғ 14 DevOps of Data: BIͷͨΊͷETL(Extract, Transform and Load)ର Ԡࣄྫɻ

    TBӽ͑ͷMySQL ڊେςʔϒϧΛ 1೔Ͱ BigQuery΁LOAD͢Δ PascalʙPuree + ngx_lua + Fluentd + BigQueryͰͭ͘ΔϝϧΧϦͷϩά෼ੳج൫ʙ
  6. 19.

    19 SREsΛߟ͑Δ " αΠϩԽʹ͍ͭͯิ଍ " DevOps " Dev + OpsͰ։ൃͱӡ༻ͷ୤αΠϩԽΛ໨ࢦ͠·͢

    " ޙʹઆ໌͢ΔMicroservicesͰ͸ɺαʔϏε͝ͱʹαΠϩԽΛ໨ࢦ͠· ͢ https://speakerdeck.com/mercari/mtc2018-microservices-platform-at-mercari
  7. 38.

    Toil limit 38 Toilͱ͸SREຊ΋ग़ͯ͘Δ֓೦ 
 ToilͷՄೳੑ͕ߴ͍࡞ۀͱ͸
 • ख࡞ۀͰ͋Δ͜ͱ • ܁Γฦ͞ΕΔ͜ͱ

    • ࣗಈԽͰ͖Δ͜ͱ • ઓज़తͰ͋Δ͜ͱ • ௕ظతͳՁ஋Λ࣋ͨͳ͍͜ͱ • αʔϏεͷ੒௕ʹରͯ͠O(n)Ͱ͋Δ͜ͱ
  8. 55.

    ઓུతSRE૊৫࿦ 55 ੒௕ظ (Growth Phase / Hyper Growth Phase) Ref.

    https://bfore.hongotechgarage.com/entry/evolution_of_a_product
  9. 58.

    ઓུతSRE૊৫࿦ 58 ੒௕ظ (Growth Phase) ͦ͜ͰɺStickiness(೪ண౓) / Install਺΍ DAUɺWAUɺ MAUɺ(e.g.

    ޿ࠂऩೖϞσϧͰ͸ແ͍৔߹͸GMV΋ؚΊ Δ)ͳͲͷKPIΛݟͯɺҰఆ਺Λ௒͑ͨΒॱௐʹ੒௕ͯ͠ ͍Δͱ൑அ͢ΔϏδωεతઓུ؍఺Λզʑ΋ҙࣝ͢Δɻ
  10. 60.

    ઓུతSRE૊৫࿦ 60 ੒௕ظ (Growth Phase) ྫ͑͹ɺ SRE͕ ”DevOps of Data”

    (e.g. ETL, pipelining)Λߦ͍BigQueryʹૹΔɻ
 
 ͦΕΛBIνʔϜ͕ɺLooker΍Slack౳Λ࢖ͬͯKPIΛݟΕΔঢ়ଶʹ͓ͯ͠ ͘ɻͳͲ https://looker.com/blog/must-have-kpis-for-your-marketing-dashboards
  11. 64.

    ઓུతSRE૊৫࿦ 64 ੒௕ظ(Growth Phase): ूத͢΂͖΋ͷ •ΦϯϘʔσΟϯά४උ •Document੔උ (Wiki, README, Src΁ͷίϝϯτ,

    DesignDoc) •Booster(ΦϦΤϯςʔγϣϯ, Kickstartܥπʔϧ, etc) •࠾༻׆ಈ •ొஃɺϒϩάɺetc
  12. 69.

    ઓུతSRE૊৫࿦: Document੔උ 69 Ducumentʹ͍ͭͯɺҎԼ͸Ͱ͖͍ͯ·͔͢? • ৘ใͷҰݩ؅ཧͱ੔ཧ • ඞཁͳ৘ใ͕ҰՕॴ͘Β͍ʹू·͍ͬͯ·͔͢ʁ • ԿՕॴ΋୳͢ͷ΋ඇޮ཰Ͱϛε͕૿͑·͢

    • e.g.ʮ˓˓͸ͲͷαʔϏεΛ࢖͑͹ྑ͍͔ʯϋοΩϦΘ͔Δ • ੔ཧ͞Ε͍ͯ·͔͢? • e.g.ʮΧςΰϥΠζ͞ΕͨπϦʔߏ଄ʯͰɺͲͷ৘ใ͕Ͳ͜ʹॻ͔Εͯ ͍Δ͔ϋοΩϦ͍ͯ͠Δɺݕࡧਫ਼౓΋ྑ͍
  13. 70.

    ઓུతSRE૊৫࿦: Document੔උ 70 Ducumentʹ͍ͭͯɺҎԼ͸Ͱ͖͍ͯ·͔͢? • ಋઢΛৗʹҙࣝ! • e.g. README΍λεΫ؅ཧνέοτʹWiki΍DesignDoc ΁ͷಋઢ(URL

    Link) ͕ுΒΕ͍ͯͯɺޙ͔ΒḷΓ΍͍͢ • Ұݩ؅ཧ͕పఈͰ͖͍ͯΕ͹ෆཁͱͳΔ͕ɺͦ͏Ͱͳ͍ ͜ͱ΋ଟ͍ͷͰɺͦͷରԠࡦͱͯ͠΋ྑ͍ํ๏
  14. 71.

    ઓུతSRE૊৫࿦: Document੔උ 71 Ducumentʹ͍ͭͯɺҎԼ͸Ͱ͖͍ͯ·͔͢? • ू໿؅ཧ • ΨΠυϥΠϯͷ࡞੒ (͜Ε΋ಋઢ) •

    awesomeԿͪΌΒͷ༷ͳrepositoryΛ࡞੒͢Δͷ΋ྑ͍ ΍Γํͩͱࢥ͍·͢ɻ • Ref: Awesome Lists
  15. 74.

    ઓུతSRE૊৫࿦: Document੔උ 74 ໋໊نଇ΍໋໊ن໿࡞Γɺߏ଄ઃܭ͸ૣ͍͏ͪʹɻ • ࠞཚΛট͘ྫ • Web V1 ͱ

    web2͕࣮͸ಉ͡ҙຯͰɺWeb V2͸ͦ ΕΒͱผɻ͔͠΋pcwebͱ͍͏ͷ΋͋Δ͕ɺͦͷ ҧ͍͸ෆ໌ɻͦΕͧΕͷಠࣗ༻ޠͷҙຯ͸terms΋ ແ͍ͷͰΘ͔Βͳ͍ঢ়ଶʹؕͬͨɻ
  16. 75.

    ઓུతSRE૊৫࿦: Document੔උ 75 ໋໊نଇ΍໋໊ن໿࡞Γɺߏ଄ઃܭ͸ૣ͍͏ͪʹɻ • ࠞཚΛট͘ྫ • prefix΍suffixͷ෇͚ํ͕ΊͪΌͪ͘ΌͰɺຖճਖ਼نදݱΛۦ࢖ͯ͠ݕ ࡧ͢ΔӋ໨ʹͳ͍ͬͯΔ (ਖ਼نදݱ͕࢖͑Δ؀ڥͳΒ·ͩྑ͍͕...)

    • ͜͏͍ͬͨ͜ͱ͸ɺෳ਺ͷνʔϜͰ࿈ܞ͢Δ͔Βඇৗʹൃੜ͠΍͍͢ • ଟ͘ͷձࣾ͸͜͏͍ͬͨ໰୊Λى͜͢ϦεΫ͕͋Δ • ͔ͩΒͦ͜஫ҙͯ͠పఈ͢Δ͜ͱ͕େࣄ
  17. 76.

    ઓུతSRE૊৫࿦: Document੔උ 76 • Ducumentʹ͍ͭͯɺҎԼ͸Ͱ͖͍ͯ·͔͢? • ໌֬Խ • RFCΛࢀߟʹ͢Δ •

    MUST (͠ͳ͚Ε͹ͳΒͳ͍), SHOULD (͢Δ΂͖), MUST NOT, SHOULD NOT ΛϋοΩϦͤ͞Δ • https://www.ietf.org/rfc/rfc2119.txt
  18. 77.

    ઓུతSRE૊৫࿦: Document੔උ 77 • จ຺ͷ༻ҙ • ԿނͦΕ͕ඞཁ͔ɺ୭͕Ͳ͏΍ͬͯ࡞Δͷ͔/࡞ͬͨͷ͔ཧղͰ͖Δ΋ͷ • DesignDocΛࢀߟʹ͢Δ •

    http://ukai.jp/Slides/2007/0907-itpro/ HackersSoftwareEngineering.pdf • Src code΁ͷίϝϯτɺ README • ඞཁ࠷௿ݶͷઆ໌΍खॱ΍ಋઢ
  19. 81.

    ઓུతSRE૊৫࿦ 81 ੒௕ظ(Growth Phase)ͷޙظ • Microservices (ུ: MS)Խ • ݱ࣌఺Ͱ͸ɺ࠷ॳ͸Monolithic

    Architecture Λ࠾༻ ͍ͯͯ͠ɺ͞ΒͳΔ੒௕ظ(௒੒௕ظ)͕࣮֬ʹདྷ ΔࣄΛೝࣝͰ͖͔ͯΒҠߦ͢ΔProduct΋ଟ͋͘Δ Ͱ͠ΐ͏
  20. 83.

    ઓུతSRE૊৫࿦ 83 ੒௕ظ(Growth Phase)ͷޙظͰɺ༏ઌతʹऔΓ૊Ή͜ͱ • ߏ੒؅ཧπʔϧͷϦϑΝΫλϦϯάͷ։࢝ • ֤πʔϧ౳ͷ൚༻ੑ௥Ճͷ։࢝ • Toil๾໓׆ಈͷ։࢝

    • ඇਪ঑ɾഇࢭ ͷపఈ
 ͜͜·ͰʹDocumentͷ༻ҙ͕ࡁΜͰ͍ͯɺΦϯϘʔσΟϯά͕εϜʔζ ʹߦ͍͑ͯͨΒɺൺֱతૣΊʹ͜ΕΒ΋։࢝͢Δ͜ͱ͕Ͱ͖Δ͸ͣͰ͢ɻ
  21. 89.

    ઓུతSRE૊৫࿦ 89 • ϝϧΧϦͷྫ • SRE team • طଘͷMonolithic architecture

    ͱͦͷMicroservices΁ͷҠߦޙͷӡ༻Λι ϑτ΢ΣΞͷٕज़Λۦ࢖ͯ͠ӡ༻͢Δ • Microservices Platform team • ઌߦͯ͠Microservicesج൫Λ࡞Δઐ໳෦ୂ • Microservices Architecture team • MS Platform ઌߦͯ͠Microservicesج൫Λ࡞Δઐ໳෦ୂ • Microservices Developers • Service ΛMicroservices ʹҠߦ͢Δ։ൃऀ
  22. 90.

    ઓུతSRE૊৫࿦ 90 • ϚΠΫϩαʔϏεΞʔΩςΫνϟΛऔΓೖΕΔ͜ͱʹΑͬͯɺ ൃੜ͢ΔτϨʔυΦϑΛཧղ͢Δ 
 ◦ “ٯίϯ΢ΣΠͷ๏ଇ͕ಇ͘͜ͱʹΑΔνʔϜͷݽཱԽͱ νʔϜؒͷίϛϡχέʔγϣϯෆ଍” ◦

    “ٕज़తͳεϓϩʔϧ(ෆنଇͳ੒௕) ʹΑΔ๲େͳίετ” ◦ “γεςϜো֐ൃੜύλʔϯͷ૿Ճ” ◦ “ٕज़ͱΠϯϑϥͷϦιʔεୣ͍߹͍”
  23. 98.

    ઓུతSRE૊৫࿦ 98 • ࠷ॳ͔ΒϚΠΫϩαʔϏε? / খ͞ͳاۀ (e.g. ελʔτΞο ϓ)ͷ৔߹
 ▪

    “৽͍͠ձࣾͷ৽͍͠ΞϓϦέʔγϣϯʹ͸ɺϚΠΫϩαʔ Ϗεʹద੾ʹ෼཭Ͱ͖Δ΄Ͳɺػೳ͕ͦ΋ͦ΋ଟ͋͘Γ· ͤΜɻ” ▪ Ҿ༻ݩ: ϓϩμΫγϣϯϨσΟϚΠΫϩαʔϏε
  24. 105.

    105 ·ͱΊΔͱ • ϚʔέςΟϯάɺϒϥϯσΟϯά • εέʔϥϏϦςΟͷඞཁ౓ • ࠾༻ɺٕज़޲্ࢥߟͳͲΤϯδχΞࢹ఺
 ͳͲͷ౤ࢿ؍఺ͱ
 •

    ϓϩμΫτͷ੒ޭঢ়گ • ༧ࢉ • ରԠՄೳͳΤϯδχΞ਺
 
 ͳͲͷࢿຊͷόϥϯε͕औΕ͍ͯΔͳΒɺ͍ͭ։࢝ͯ͠΋ྑ͍ͱࢥ͍·͢!
  25. 108.

    108 [ิ଍] Service-oriented architecture(SOA)? • ʮͰ͖ΔݶΓڞ༗ʯʮϏδωεػೳͷ࠶ར༻ʯΛॏཁͱ͍ͯ͠ Δ • 4छྨͷαʔϏεʹ෼͔ΕΔ •

    ϏδωεɺΤϯλʔϓϥΠζɺΞϓϦέʔγϣϯɺΠϯϑϥ • ڞ௨ͷ౷࣏ͱج४ > ࣗ༝౓ • ैདྷͷRDBMSΛΑ͘࢖͏ • ݱࡏ͸ɺDevOps / CD౳͸ීٴ࢝͠Ίɻ·ͩओྲྀͳ΋ͷ͸ଘࡏ ͍ͯ͠ͳ͍
  26. 109.

    109 [ิ଍] Service-oriented architecture(SOA)? • SOAͷαʔϏεཻ౓ • αʔϏείϯϙʔωϯτͷαΠζ͸ɺখن໛ͷΞϓϦέʔγϣϯαʔ Ϗε͔Βඇৗʹେن໛ͳΤϯλʔϓϥΠζαʔϏε·ͰɺରԠ͕ग़ དྷ·͢ɻ

    • ࣮ࡍʹ͸ɺେن໛ͳ੡඼΍αϒγεςϜʹ૬౰͢Δେ͖͞ͷαʔϏ είϯϙʔωϯτΛSOA಺Ͱ࣋ͭͷ͕Ұൠతɻ • αʔϏείϯϙʔωϯτͷڞ༗͕ॏཁͳ֓೦ͷ̍ͭ • ϦϞʔτΞΫηεϓϩτίϧͱͯ͠ϝοηʔδϯάʢAMQPɺ MSMQʣ͓ΑͼSOAPʹґଘ
  27. 112.
  28. 113.

    SREۀ຿Ͱ΋औΓೖΕͨํ͕ྑ͍จԽ 113 " Մಡੑͷҙࣝ / ϦʔμϒϧίʔσΟϯά " e.g. આ໌ม਺ "

    ม਺໊ʹҙຯΛ࣋ͨͤΔ " ม਺໊ΛݟΕ͹ɺͦΕ͕ԿΛ͍ͯ͠Δ͔ཧղͰ͖Δ " ίϝϯτͷॻ͖ํ " Մಡੑͷྑ͍ؔ਺ͷॻ͖ํ " ಡΈ΍͘͢؆ܿʹॻ͚Δ༷ʹ͢ΔςΫχοΫ " อकੑ΍ߋ৽଎౓Λ্͛ɺকདྷͷࣗ෼ͷෛ୲΋Լ͛Δ " ͱͯ΋ॏཁͳ͜ͱ͕٧·͍ͬͯΔ " ڞ௨ೝࣝΛ࣋ͨͳ͍ಠུࣗޠͷ࢖༻ېࢭ
  29. 122.

    122 P-R͸ग़དྷΔ͚ͩখ͘͞ऴΘΒͤΔͱ͜Ε͚ͩಘ " Review͞Ε΍͘͢ͳΔ " Reviewerͷෛ୲͕ݮΒͤΔ " Merged Cycle͕ૣ·Δ "

    ਝ଎ͳ෮چରԠ͕ग़དྷΔ༷ʹͳΔ " Conflict΋ݮগ͢Δ " Deploy / ReleaseͷӨڹΛখ͘͞Ͱ͖Δ
 ◦ ݁Ռɺ৴པੑΛ޲্Ͱ͖ΔՄೳੑ
  30. 126.

    126 P-R͸খ͘͢͞ΔࣄΛਪ঑͍ͯ͠ΔهࣄͷҰྫ " https://www.madetech.com/blog/deployment-by-pull-requests " Feedback is easier to gather

    if the pull request is small, so some effort should be spent to make more concise pull requests. " Reducing conflicts " The pull requests are kept open just for a small amount of time, to get enough reviews and merged as quickly as possible. In order to support this workflow, the pull requests are kept as small as possible on purpose so they can be reviewed easily and merged quicker. Multiple pull requests can be opened if the work to be carried is too large to fit into a single one. " https://www.thedroidsonroids.com/blog/splitting-pull-request " There should only be one issue resolved with each pull request, and only one pull request required to resolve an issue. " https://d83tozu1c8tt6.cloudfront.net/media/resources/SDL_GitHub_BestPractices.pdf " https://blog.nathanaelcherrier.com/en/advice-great-code-review/ " A developer who wants to receive useful review comments or wants these colleagues to quickly review his pull request, should follow this tiny single rule: one feature = one pull request. " Reviewers are also developers who have tasks to do. Reviewing a pull request takes time. As a reviewer, when I see a big pull request which contains a lot of features I don't want to review it right now but later because I know I am going to spend a lot of time on this pull request.
  31. 127.

    [ิ଍] P-R΍ReviewͰΑ͘ར༻͢ΔҰൠతͳུޠ 127 • WIP • ࡞ۀ్தͷ৔߹ʹ࢖͏ • e.g. [WIP]

    Add an Ansible role for *** • RFC • ҙݟΛٻΊΔ • e.g. RFC label΍ɺ[RFC] Add exponential backoff with retry in example-controller • IMO • ReviewίϝϯτͰݸਓతҙݟͱ͍͏ͷΛڧௐ • .e.g. ʮ[IMO] ΠΠω :+1: ! ͋ͱɺࢲ͸͜ΕΛ෇͚Ճ͑Δͱྑͦ͞͏ͱࢥ͏Μ͚ͩ ͲɺͲ͏ࢥ͍·͔͢?ʯ
  32. 128.
  33. 129.

    ·ͱΊ 129 • Ϗδωε·Ͱࢹ໺֯Λ޿͛ͯઓུతʹ! • SRE΋ϏδωεશମΛݟΔඞཁ͕͋ΔΤϯδχΞͱཧղ͢Δ • ঢ়گʹ߹Θͤͨ৬຿ͱ੹຿Λߟ࡯͢Δඞཁ͕͋Δ • ϓϩμΫτ/Ϗδωεͷ੒௕/੒ޭ͕Ұ൪

    • ࠓͷ࣌୅΋Document͸େࣄ! • Microservices͸ಋೖܭըΛ͔ͬ͠Γͱ! • ྑ͍ͱࢥͬͨจԽ͸औΓೖΕͯΈΔ • ࣌୅͸ɺϚΠΫϩʓʓʓ / Microdevelopments !!
  34. 130.

    ࠷ޙʹ 130 ࠓ೔ͷ࿩Λฉ͍ͯ/ಡΜͰ ਂ͍ཧղͱڞײ + ڧ͍ڵຯ + ৘೤ => Ұॹʹಇ͖͍ͨ

    ͱࢥͬͯ͘Εͨ͋ͳͨ!! SRE EMΛܹืूதͰ͢! TwitterͰktykogm·ͰDM͍ͩ͘͞ or ࠙਌ձ౳Ͱ΋͓࿩͠·͠ΐ͏ɻ ҿΈͰ΋༠͍ͬͯͩ͘͞ʂʂ
  35. 131.