Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Djangoアプリケーションにおけるトイル撲滅戦記.pdf

Yasuaki Matsuda
September 18, 2018

 Djangoアプリケーションにおけるトイル撲滅戦記.pdf

Yasuaki Matsuda

September 18, 2018
Tweet

Other Decks in Programming

Transcript

  1. DjangoΞϓϦέʔγϣϯʹ͓͚Δ
    τΠϧ๾໓ઓه

    View Slide

  2. ࣗݾ঺հ
    দా ߁ষ (Yasuaki Matsuda)
    @yacchin0101
    גࣜձࣾΞΠϦοδ
    αʔόαΠυΤϯδχΞ

    View Slide

  3. ࣗݾ঺հ

    View Slide

  4. ͲΜͳਓ͕ର৅ʁ
    τΠϧ๾໓ʹڵຯ͕͋Δ
    ݱ৔ͰDjangoΛ࢖ͬͯΔ
    ೔ʑͷӡ༻ۀ຿Ͱർฐ͍ͯ͠Δਓ

    View Slide

  5. ໨࣍
    લ࢙
    τΠϧ๾໓
    τΠϧͱ͸
    ޮՌ
    ·ͱΊ

    View Slide

  6. େมਃ͠༁͍͟͝·ͤΜ

    τΠϧΛ๾໓ͨ͠Θ͚Ͱ͸ͳ͍

    View Slide

  7. େมਃ͠༁͍͟͝·ͤΜ

    SRE͸ʮτΠϧͷ๾໓ʯ͕ϝΠϯ

    View Slide

  8. τΠϧͱ͸
    τΠϧʹ͍ͭͯͷ঺հ
    photo by Yamato Fukui

    View Slide

  9. SREͱ͸
    Google͕ࣾఏএɾ࣮ફ͍ͯ͠Δ
    γεςϜ؅ཧͱαʔϏεӡ༻ͷํ๏࿦

    View Slide

  10. SREͱ͸
    IUUQTMBOEJOHHPPHMFDPNTSFCPPLJOEFYIUNM
    5ষ τΠϧͷ๾໓
    τΧήʁ
    SRE
    αΠτϦϥΠΞϏϦςΟ
    ΤϯδχΞϦϯά

    View Slide

  11. τΠϧͱ͸
    ຊ൪αʔϏεʹؔ͢Δ࡞ۀͰ
    ख࡞ۀͰ܁Γฦ͠ߦΘΕ
    ࣗಈԽ͢Δ͜ͱ͕ՄೳͰ͋Γ
    ઓज़తͰ௕ظతͳՁ஋Λ࣋ͨͣ
    ࡞ۀྔ͕αʔϏεͷ੒௕ʹൺྫ͢Δ

    View Slide

  12. ৯ثͷखચ͍
    ਎ۙͳτΠϧ

    View Slide

  13. ਎ۙͳτΠϧ
    ख࡞ۀͰ͋Δ
    ܁Γฦ͞ΕΔ
    ࣗಈԽͰ͖Δ
    ઓज़తͰͳ͍
    ௕ظతͳՁ஋Λ࣋ͨͳ͍
    αʔϏε੒௕ʹରͯ͠O(n)Ͱͳ͍
    ৯ثͷखચ͍

    View Slide

  14. ਎ۙͳτΠϧ
    શࣗಈ৯ચػ

    View Slide

  15. τΠϧ͕΋ͨΒ͢΋ͷ
    ҰॠͰՔಇΛຒΊਚ͘͞ΕΔ
    ϔΠτΛు͘Α͏ʹͳΔ
    ΩϟϦΞ͕ఀ଺͢Δ
    ͳͲ

    View Slide

  16. ·ͱΊ
    τΠϧΛ๾໓ͯ͠
    PythonͰΠϊϕʔςΟϒͳ
    ࢓ࣄΛ͠Α͏ʂ

    View Slide

  17. લ࢙
    τΠϧ๾໓લͷDjangoΞϓϦέʔγϣϯʹ͍ͭͯͷ঺հ
    photo by Yamato Fukui

    View Slide

  18. લ࢙
    Online to Offline

    View Slide

  19. લ࢙
    དྷళൢଅࢪࡦ
    Push௨஌
    ϙΠϯτ
    Ϋʔϙϯ
    ήʔϜ

    View Slide

  20. Push௨஌
    ͓ɺ෰ങ͍ʹ
    ߦ͜͏͔ͳɻ
    ΫϦΞϥϯεηʔϧ
    ։࠵தʂ

    View Slide

  21. ϙΠϯτ
    ϙΠϯτ໯͑Δ͠
    ߦ͜͏͔ͳɻ
    དྷళϙΠϯτ
    Ωϟϯϖʔϯ࣮ࢪதʂ

    View Slide

  22. Ϋʔϙϯ
    ͋ͷళͷΫʔϙϯ
    ͋Δ͡ΌΜɻ
    10%OFFΫʔϙϯ
    ഑෍தʂ

    View Slide

  23. ήʔϜ
    ΍ͬͨͥ
    ͓ΊͰͱ͏ʂ
    100PTήοτʂ

    View Slide

  24. લ࢙
    hogeΞϓϦ
    Push௨஌
    ϙΠϯτ
    Ϋʔϙϯ
    ήʔϜ
    ܾࡁ

    View Slide

  25. Offline
    ͓ళͷηʔϧ৘ใΛݟΔ
    ϙΠϯτ࢒ߴΛ֬ೝ͢Δ
    ήʔϜΛ࣮ࢪͯ͠ϙΠϯτ֫ಘ

    View Slide

  26. Online
    ͓ళͰΫʔϙϯΛ࢖͏
    ΞϓϦͰܾࡁ͢Δ

    View Slide

  27. લ࢙
    hogeΞϓϦͱࢲͷؔ܎
    αʔόαΠυػೳ࣮૷
    Πϯϑϥߏங
    ӡ༻ɾอक୲౰ऀ

    View Slide

  28. લ࢙
    CPU࢖༻཰
    Push௨஌

    View Slide

  29. લ࢙
    ֎஫࣌୅ ಺੡࣌୅ τΠϧ๾໓࣌୅
    2015೥ 2016೥ 2018೥

    View Slide

  30. ֎஫࣌୅
    ελʔτΞοϓاۀ
    ։ൃऀෆ଍
    ֎஫

    View Slide

  31. ʢ2015.x.xʣ
    ߏ੒
    App & Batch Server
    Admin Server
    Coupon Server
    ELB
    ELB

    View Slide

  32. ֎஫࣌୅
    ຯΘ͍ਂ͍ߏ੒
    DBಉډ໰୊
    SSLূ໌ॻ௚ஔ͖໰୊
    σϓϩΠ: SCP
    ͳͲ

    View Slide

  33. ઌํʮΞϓϦͷػೳ֦ு͍ͨ͠ʯ
    ֎஫࣌୅
    Ӧۀʮঝ஌͍ͨ͠·ͨ͠ʂʯ
    ೖࣾͨ͠দాʮΑΖ͓͘͠Ͷ͕͍͠·͢ʂʯ
    PMʮࠓޙͷ֦ுΛߟ͑ͯ಺੡͢Δ͔ʯ

    View Slide

  34. ಺੡࣌୅
    ࣾ಺։ൃ
    Python / Django Ͱ։ൃ
    GitLabͰιʔε؅ཧ
    AWSͰߏங

    View Slide

  35. ʢ2016.x.xʣ
    ߏ੒
    App Server
    Admin Server
    Coupon Server RDS
    ELB
    ELB
    Batch Server

    View Slide

  36. ಺੡࣌୅
    ଟগ͸վળ͞Εͨ
    RDS(ϚωʔδυαʔϏε)
    PHP͔ΒDjango΁Ҡߦ
    σϓϩΠ: Git Pull
    ͳͲ

    View Slide

  37. ϩʔϯνޙ
    ઌํʮ͜͜ͷจݴमਖ਼Λ͓ئ͍Ͱ͖·͢ʁʯ
    PMʮঝ஌͠·ͨ͠ɻদా͞Μ͓ئ͍͠·͢ɻʯ
    দాʮ͸͍ɻʯ

    View Slide

  38. ಺੡࣌୅
    ͋Εʁ
    ʮ͜ͷલ΋ґཔ͞ΕͨΑ͏ͳʯ
    ʮ࠷ۙɺ։ൃͰ͖͍ͯͳ͍ʯ
    ʮਓ͕଍Γͳ͍ʯ

    View Slide

  39. ಺੡࣌୅
    τΠϧʹ૶ૡ͖ۤ͠Ή࣌୅ͷ౸དྷ

    View Slide

  40. τΠϧ๾໓
    τΠϧ๾໓ͷաఔʹ͍ͭͯ঺հ
    photo by Yamato Fukui

    View Slide

  41. ํ਑
    Twelve-Factor AppΛ
    Managed ServiceΛ
    CI / CD

    View Slide

  42. ํ਑

    View Slide

  43. Twelve-Factor App
    ࣌ؒɾίετͷ࠷খԽ
    Ҡ২ੑ޲্
    Πϯϑϥ؅ཧ͔Βͷ։์
    ܧଓతσϓϩΠ
    εέʔϧΞοϓ
    IUUQTGBDUPSOFU

    View Slide

  44. Managed Service
    Πϯϑϥ؅ཧෆཁ
    ӡ༻ϦεΫ௿ݮ
    ܧଓతվળ

    View Slide

  45. CI / CD
    ख࡞ۀ͔Βͷ։์
    ඼࣭޲্

    View Slide

  46. τΠϧ๾໓
    Case 1

    View Slide

  47. τΠϧ๾໓લͷঢ়گ
    ؀ڥߏங࡞ۀ
    ৽͍͠ΤϯδχΞ͕ࢀը
    README௨Γʹ؀ڥߏங
    ಈ͔ͳ͍
    ৬ਓͷग़ਞ

    View Slide

  48. ػೳ։ൃ։࢝·Ͱʹ͔͔࣌ؒΔ
    ৬ਓͷՔಇ͕औΒΕΔ
    τΠϧ๾໓લͷঢ়گ
    ՝୊

    View Slide

  49. τΠϧ๾໓ઓུ
    Docker Compose

    View Slide

  50. τΠϧ๾໓ઓུ
    ίϯςφܕԾ૝؀ڥ
    ؀ڥࠩҟ͕খ͘͞ͳΔ
    Ҡ২ੑ͕ߴ͍
    Docker

    View Slide

  51. τΠϧ๾໓ઓུ
    Docker Compose
    ෳ਺ίϯςφΛఆٛ
    ෳ਺ίϯςφΛ࣮ߦ
    docker-compose up ͷΈ

    View Slide

  52. ͙͢։ൃͰ͖ΔΑ͏ʹͳͬͨ
    ৬ਓͷग़ਞ͕཈͑ΒΕͨ
    ؀ڥࠩҟʹΑΔόά͕཈͑ΒΕͨ
    τΠϧ๾໓ޙ
    վળޮՌ

    View Slide

  53. τΠϧ๾໓ޙ

    View Slide

  54. τΠϧ๾໓
    Case 2

    View Slide

  55. खॱॻͷ࡞੒
    1୆ͣͭॖୀɾߋ৽
    νΣοΧʔͱෳ਺ਓମ੍
    σϓϩΠ࡞ۀ
    τΠϧ๾໓લͷঢ়گ

    View Slide

  56. σϓϩΠखॱ൥ࡶʹΑΔ
    ώϡʔϚϯΤϥʔϦεΫ
    σϓϩΠʹର͢Δ޻਺͕ߴΊ
    τΠϧ๾໓લͷঢ়گ
    ՝୊

    View Slide

  57. τΠϧ๾໓ઓུ
    GitLab CI
    Amazon ECS

    View Slide

  58. τΠϧ๾໓ઓུ
    ίϯςφΦʔέετϨʔγϣϯ
    αʔό؅ཧෆཁ
    εέδϡʔϧ࣮ߦ
    Amazon ECS

    View Slide

  59. τΠϧ๾໓ઓུ
    GitϦϙδτϦ
    CI / CDػೳ
    issue؅ཧ
    ίʔυϨϏϡʔ
    GitLab

    View Slide

  60. RDS
    ELB
    X-Ray
    ECS
    App Server Service
    X-Ray
    Batch Task
    ELB
    ELB
    ɾɾɾ
    X-Ray
    X-Ray
    Coupon Server Service
    ɾɾɾ
    X-Ray
    X-Ray
    Admin Server Service
    ɾɾɾ
    X-Ray
    CloudWatch
    Logs
    S3
    τΠϧ๾໓ઓུ

    View Slide

  61. ιʔεमਖ਼
    ϨϏϡʔ
    ϙνΔ
    τΠϧ๾໓ޙ
    σϓϩΠखॱ

    View Slide

  62. ਓҝతϛε͕ൃੜ͠ͳ͘ͳͬͨ
    σϓϩΠʹର͢Δ޻਺͕ݮͬͨ
    վળޮՌ
    τΠϧ๾໓ޙ

    View Slide

  63. /hoge/ͰPOST࣌͸IP੍ݶ͍ͨ͠
    τΠϧ๾໓ޙ

    View Slide

  64. τΠϧ๾໓ޙ
    SQLΫΤϦվળ

    View Slide

  65. ECR
    X-Ray
    APP Server Service
    X-Ray
    New Task
    Old Task
    1. Image Build & Push New Image
    2. Create New Task Definition
    3. Update Service
    1. Push Source Code
    X-Ray
    App Task: 1
    X-Ray
    App Task: 2
    GitLab
    2. Create Merge Request(MR)
    3. Review & LGTM & Accept MR
    τΠϧ๾໓ޙ

    View Slide

  66. τΠϧ๾໓
    ͓·͚

    View Slide

  67. τΠϧ๾໓લͷঢ়گ
    ੩తϑΝΠϧσϓϩΠ
    ιʔεमਖ਼
    ϨϏϡʔ
    ϙνΔ

    View Slide

  68. ɻΛ෇༩͍ͨ͠
    τΠϧ๾໓ޙ

    View Slide

  69. 1. npm install & build
    2. Put Artifacts to S3 Bucket
    1. Push Source Code
    GitLab
    2. Create Merge Request(MR)
    3. Review & LGTM & Accept MR
    AWS CLI
    τΠϧ๾໓ޙ

    View Slide

  70. τΠϧ๾໓
    ߋʹ͓·͚

    View Slide

  71. Ϣχοτςετ
    ίʔσΟϯάن໿֬ೝ
    ෳࡶ౓֬ೝ
    τΠϧ๾໓લͷঢ়گ
    ιʔείʔυϨϏϡ࡞ۀ

    View Slide

  72. 1. Unit Test
    2. Flake8
    3. Radon
    GitLab
    ςετ࣮ࢪʹΑΔ඼࣭୲อ
    ίʔσΟϯάن໿ͷ֬ೝ
    ίʔυͷෳࡶ౓Λ֬ೝ
    1. Push Source Code
    2. Create Merge Request(MR)
    3. Review & LGTM & Accept MR
    τΠϧ๾໓ޙ

    View Slide

  73. τΠϧ๾໓ޙ

    View Slide

  74. τΠϧ๾໓
    ͓·͚
    (͜ΕͰ࠷ޙ)

    View Slide

  75. τΠϧ๾໓લͷঢ়گ
    όοναʔόෆཁ
    ECSͰఆظతʹ࣮ߦͰ͖Δ
    ࢖ͬͨ෼͚ͩͷྉۚ

    View Slide

  76. τΠϧ๾໓ޙ

    View Slide

  77. τΠϧ๾໓
    Case 3

    View Slide

  78. τΠϧ๾໓લͷঢ়گ
    ֤αʔό͔ΒΞΫηεϩάऔಘ
    grepɾawk৬ਓ
    ಛఆϢʔβͷΞΫηεௐࠪ

    View Slide

  79. ௐࠪʹ͕͔͔࣌ؒΔ
    ώϡʔϚϯΤϥʔʹΑΔϦεΫ
    τΠϧ๾໓લͷঢ়گ
    ՝୊

    View Slide

  80. τΠϧ๾໓ઓུ
    CloudWatch Logs

    View Slide

  81. τΠϧ๾໓ઓུ
    ϩάͷ؂ࢹ
    ϩάͷอଘ
    ϩά΁ͷΞΫηε
    CloudWatch Logs

    View Slide

  82. τΠϧ๾໓ઓུ
    settings.py

    View Slide

  83. τΠϧ๾໓ઓུ
    docker-compose.yml

    View Slide

  84. ਺෼Ͱௐࠪ׬ྃ
    ਓҝతϛεʹΑΔαʔϏεఀࢭ͕ͳ͍
    ඇΤϯδχΞͰ΋ௐࠪՄೳ
    τΠϧ๾໓
    վળޮՌ

    View Slide

  85. τΠϧ๾໓ޙ
    ϩάΛҰՕॴʹू໿
    લํҰகݕࡧ
    ೔࣌ݕࡧ

    View Slide

  86. τΠϧ๾໓ޙ

    View Slide

  87. τΠϧ๾໓
    Case 4

    View Slide

  88. τΠϧ๾໓લͷঢ়گ
    RDSεέʔϧΞοϓ࡞ۀ
    ࣮ߦ͞ΕΔSQL͕ѱ͍
    CPU࢖༻཰100%ுΓ෇͖
    αʔϏεμ΢ϯൃੜ
    ۚͰղܾ

    View Slide

  89. αʔϏεμ΢ϯͷՄೳੑ
    αʔϏεҡ࣋ʹίετ͕͔͔Δ
    ࣮૷༧ఆͩͬͨ΋ͷ͕஗Ԇ
    τΠϧ๾໓લͷঢ়گ
    ՝୊

    View Slide

  90. τΠϧ๾໓ઓུ
    Django QuerySet

    View Slide

  91. σʔλϕʔεͷ҆ఆՔಇ
    ༨ܭͳεέʔϧΞοϓΛ཈͑Δ
    վળޮՌ
    τΠϧ๾໓ޙ

    View Slide

  92. ήʔϜ࣮ࢪՄ൱API
    ήʔϜ͸1೔1ճ·Ͱ
    ΞϓϦશମͰ1೔ʹ10,000ճ
    τΠϧ๾໓ޙ

    View Slide

  93. τΠϧ๾໓ޙ

    View Slide

  94. τΠϧ๾໓ޙ

    View Slide

  95. PushͰήʔϜར༻ऀ૿Ճ
    ϨεϙϯεྼԽ
    ΠϯελϯεεέʔϧΞοϓ
    τΠϧ๾໓ޙ

    View Slide

  96. τΠϧ๾໓ޙ

    View Slide

  97. τΠϧ๾໓ޙ

    View Slide

  98. τΠϧ๾໓ޙ

    View Slide

  99. τΠϧ๾໓
    Case 5

    View Slide

  100. τΠϧ๾໓લͷঢ়گ
    ؅ཧαʔόʹϩάΠϯͰ͖ͳ͍໰୊ͷௐࠪ
    ϩάௐࠪ
    ϩάʹు͔Ε͍ͯͳ͍
    ιʔεΛݟΔ

    View Slide

  101. ໰୊ղܾʹ͕͔͔࣌ؒΔ
    τΠϧ๾໓લͷঢ়گ
    ՝୊

    View Slide

  102. τΠϧ๾໓લͷঢ়گ

    View Slide

  103. τΠϧ๾໓લͷঢ়گ
    Effective Python

    Item 14: Prefer Exceptions to returning None
    νϟϦ
    Effective
    Python

    View Slide

  104. τΠϧ๾໓લͷঢ়گ
    ྑ͘ͳ͍఺
    except۟ͰϩΪϯά͍ͯ͠ͳ͍
    except۟ͰNoneΛฦ٫

    View Slide

  105. τΠϧ๾໓ઓུ
    Django Logging

    View Slide

  106. ϩάΛݟΔ͚ͩͰঢ়گ೺Ѳ
    τΠϧ๾໓
    վળޮՌ

    View Slide

  107. τΠϧ๾໓ޙ

    View Slide

  108. ྑ͘ͳͬͨ఺
    ϩάͰঢ়گΛ೺ѲͰ͖Δ
    ಛผͳྫ֎Λ౤͛Δ͜ͱͰ
    ΍Γ͍ͨ͜ͱΛ໌֬ʹͨ͠
    τΠϧ๾໓ޙ

    View Slide

  109. τΠϧ๾໓
    Case 6

    View Slide

  110. τΠϧ๾໓લͷঢ়گ
    ֤APIͷϨΠςϯγௐࠪ
    ֤αʔό͔ΒΞΫηεϩάऔಘ
    grepɾawk৬ਓ

    View Slide

  111. ͕͔͔࣌ؒΔ
    ϦΞϧλΠϜʹΘ͔Βͳ͍
    τΠϧ๾໓લͷঢ়گ
    ՝୊

    View Slide

  112. τΠϧ๾໓ઓུ
    AWS X-Ray

    View Slide

  113. τΠϧ๾໓ઓུ
    ϦΫΤετʹؔ͢Δσʔλऩू
    σʔλͷදࣔɾϑΟϧλϦϯά
    தؒͷαʔϏεৄࡉ΋දࣔ
    AWS X-Ray

    View Slide

  114. τΠϧ๾໓ઓུ
    X-Ray
    X-Ray
    Daemon

    View Slide

  115. τΠϧ๾໓ઓུ
    settings.py

    View Slide

  116. τΠϧ๾໓ઓུ
    Dockerfile

    View Slide

  117. ຊ൪؀ڥͷঢ়گ͕೺ѲՄೳ
    ύϑΥʔϚϯενϡʔχϯά˕
    ো֐ରԠͰ໾ʹཱͭ
    վળޮՌ
    τΠϧ๾໓ޙ

    View Slide

  118. τΠϧ๾໓ޙ

    View Slide

  119. τΠϧ๾໓ޙ

    View Slide

  120. τΠϧ๾໓ޙ

    View Slide

  121. τΠϧ๾໓ޙ

    View Slide

  122. ޮՌ
    τΠϧ๾໓ʹΑΔޮՌͷ঺հ
    photo by Yamato Fukui

    View Slide

  123. ޮՌ
    ো֐ରԠ͕ݮͬͨ
    ख࡞ۀݮʹ൐͏ώϡʔϚϯΤϥʔ͕গͳ͘ͳͬͨ͜ͱ͕େ͖͍
    ΤϯδχΞϦϯάʹׂ͚Δ͕࣌ؒ૿͑ͨ
    DjangoΞϓϦέʔγϣϯ։ൃʹ஫ྗͰ͖ΔΑ͏ʹͳͬͨ
    ΩϟϦΞΞοϓͰ͖ͨ
    ϦʔμʹͳΓ·ͨ͠

    View Slide

  124. ·ͱΊ
    photo by Yamato Fukui

    View Slide

  125. ·ͱΊ
    ʮͦΕτΠϧ͡ΌͶʁʯͷਫ਼ਆ
    τΠϧΛݮΒͤ͹ΤϯδχΞϦϯάʹूதͰ͖Δ
    ܧଓͯ͠վળͰ͖Δঢ়ଶΛ࡞Δ
    τΠϧ͸ແݶʹ༙͖ଓ͚Δ
    Managed ServiceɾCI / CDͷ׆༻
    ؅ཧର৅ΛݮΒ͠ɺࣗಈԽΛਪਐ͢Δ

    View Slide

  126. ·ͱΊ
    τΠϧΛ๾໓ͯ͠
    PythonͰΠϊϕʔςΟϒͳ
    ࢓ࣄΛ͠Α͏ʂ

    View Slide

  127. ࢀߟจݙ
    O'Reilly Japan - SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά
    O'Reilly Japan - Effective Python

    View Slide