Djangoアプリケーションにおけるトイル撲滅戦記.pdf

https://pycon.jp/2018/

Transcript

1. DjangoΞϓϦέʔγϣϯʹ͓͚Δ
   τΠϧ๾໓ઓه
   

   View Slide

2. ࣗݾ঺հ
   দా ߁ষ (Yasuaki Matsuda)
   @yacchin0101
   גࣜձࣾΞΠϦοδ
   αʔόαΠυΤϯδχΞ
   

   View Slide

3. ࣗݾ঺հ
   

   View Slide

4. ͲΜͳਓ͕ର৅ʁ
   τΠϧ๾໓ʹڵຯ͕͋Δ
   ݱ৔ͰDjangoΛ࢖ͬͯΔ
   ೔ʑͷӡ༻ۀ຿Ͱർฐ͍ͯ͠Δਓ
   

   View Slide

5. ໨࣍
   લ࢙
   τΠϧ๾໓
   τΠϧͱ͸
   ޮՌ
   ·ͱΊ
   

   View Slide

6. େมਃ͠༁͍͟͝·ͤΜ
   
   τΠϧΛ๾໓ͨ͠Θ͚Ͱ͸ͳ͍
   
   

   View Slide

7. େมਃ͠༁͍͟͝·ͤΜ
   
   SRE͸ʮτΠϧͷ๾໓ʯ͕ϝΠϯ
   
   

   View Slide

8. τΠϧͱ͸
   τΠϧʹ͍ͭͯͷ঺հ
   photo by Yamato Fukui
   

   View Slide

9. SREͱ͸
   Google͕ࣾఏএɾ࣮ફ͍ͯ͠Δ
   γεςϜ؅ཧͱαʔϏεӡ༻ͷํ๏࿦
   

   View Slide

10. SREͱ͸
    IUUQTMBOEJOHHPPHMFDPNTSFCPPLJOEFYIUNM
    5ষ τΠϧͷ๾໓
    τΧήʁ
    SRE
    αΠτϦϥΠΞϏϦςΟ
    ΤϯδχΞϦϯά
    

    View Slide

11. τΠϧͱ͸
    ຊ൪αʔϏεʹؔ͢Δ࡞ۀͰ
    ख࡞ۀͰ܁Γฦ͠ߦΘΕ
    ࣗಈԽ͢Δ͜ͱ͕ՄೳͰ͋Γ
    ઓज़తͰ௕ظతͳՁ஋Λ࣋ͨͣ
    ࡞ۀྔ͕αʔϏεͷ੒௕ʹൺྫ͢Δ
    

    View Slide

12. ৯ثͷखચ͍
    ਎ۙͳτΠϧ
    

    View Slide

13. ਎ۙͳτΠϧ
    ख࡞ۀͰ͋Δ
    ܁Γฦ͞ΕΔ
    ࣗಈԽͰ͖Δ
    ઓज़తͰͳ͍
    ௕ظతͳՁ஋Λ࣋ͨͳ͍
    αʔϏε੒௕ʹରͯ͠O(n)Ͱͳ͍
    ৯ثͷखચ͍
    

    View Slide

14. ਎ۙͳτΠϧ
    શࣗಈ৯ચػ
    

    View Slide

15. τΠϧ͕΋ͨΒ͢΋ͷ
    ҰॠͰՔಇΛຒΊਚ͘͞ΕΔ
    ϔΠτΛు͘Α͏ʹͳΔ
    ΩϟϦΞ͕ఀ଺͢Δ
    ͳͲ
    

    View Slide

16. ·ͱΊ
    τΠϧΛ๾໓ͯ͠
    PythonͰΠϊϕʔςΟϒͳ
    ࢓ࣄΛ͠Α͏ʂ
    

    View Slide

17. લ࢙
    τΠϧ๾໓લͷDjangoΞϓϦέʔγϣϯʹ͍ͭͯͷ঺հ
    photo by Yamato Fukui
    

    View Slide

18. લ࢙
    Online to Offline
    

    View Slide

19. લ࢙
    དྷళൢଅࢪࡦ
    Push௨஌
    ϙΠϯτ
    Ϋʔϙϯ
    ήʔϜ
    

    View Slide

20. Push௨஌
    ͓ɺ෰ങ͍ʹ
    ߦ͜͏͔ͳɻ
    ΫϦΞϥϯεηʔϧ
    ։࠵தʂ
    

    View Slide

21. ϙΠϯτ
    ϙΠϯτ໯͑Δ͠
    ߦ͜͏͔ͳɻ
    དྷళϙΠϯτ
    Ωϟϯϖʔϯ࣮ࢪதʂ
    

    View Slide

22. Ϋʔϙϯ
    ͋ͷళͷΫʔϙϯ
    ͋Δ͡ΌΜɻ
    10%OFFΫʔϙϯ
    ഑෍தʂ
    

    View Slide

23. ήʔϜ
    ΍ͬͨͥ
    ͓ΊͰͱ͏ʂ
    100PTήοτʂ
    

    View Slide

24. લ࢙
    hogeΞϓϦ
    Push௨஌
    ϙΠϯτ
    Ϋʔϙϯ
    ήʔϜ
    ܾࡁ
    

    View Slide

25. Offline
    ͓ళͷηʔϧ৘ใΛݟΔ
    ϙΠϯτ࢒ߴΛ֬ೝ͢Δ
    ήʔϜΛ࣮ࢪͯ͠ϙΠϯτ֫ಘ
    

    View Slide

26. Online
    ͓ళͰΫʔϙϯΛ࢖͏
    ΞϓϦͰܾࡁ͢Δ
    

    View Slide

27. લ࢙
    hogeΞϓϦͱࢲͷؔ܎
    αʔόαΠυػೳ࣮૷
    Πϯϑϥߏங
    ӡ༻ɾอक୲౰ऀ
    

    View Slide

28. લ࢙
    CPU࢖༻཰
    Push௨஌
    

    View Slide

29. લ࢙
    ֎஫࣌୅ ಺੡࣌୅ τΠϧ๾໓࣌୅
    2015೥ 2016೥ 2018೥
    

    View Slide

30. ֎஫࣌୅
    ελʔτΞοϓاۀ
    ։ൃऀෆ଍
    ֎஫
    

    View Slide

31. ʢ2015.x.xʣ
    ߏ੒
    App & Batch Server
    Admin Server
    Coupon Server
    ELB
    ELB
    

    View Slide

32. ֎஫࣌୅
    ຯΘ͍ਂ͍ߏ੒
    DBಉډ໰୊
    SSLূ໌ॻ௚ஔ͖໰୊
    σϓϩΠ: SCP
    ͳͲ
    

    View Slide

33. ઌํʮΞϓϦͷػೳ֦ு͍ͨ͠ʯ
    ֎஫࣌୅
    Ӧۀʮঝ஌͍ͨ͠·ͨ͠ʂʯ
    ೖࣾͨ͠দాʮΑΖ͓͘͠Ͷ͕͍͠·͢ʂʯ
    PMʮࠓޙͷ֦ுΛߟ͑ͯ಺੡͢Δ͔ʯ
    

    View Slide

34. ಺੡࣌୅
    ࣾ಺։ൃ
    Python / Django Ͱ։ൃ
    GitLabͰιʔε؅ཧ
    AWSͰߏங
    

    View Slide

35. ʢ2016.x.xʣ
    ߏ੒
    App Server
    Admin Server
    Coupon Server RDS
    ELB
    ELB
    Batch Server
    

    View Slide

36. ಺੡࣌୅
    ଟগ͸վળ͞Εͨ
    RDS(ϚωʔδυαʔϏε)
    PHP͔ΒDjango΁Ҡߦ
    σϓϩΠ: Git Pull
    ͳͲ
    

    View Slide

37. ϩʔϯνޙ
    ઌํʮ͜͜ͷจݴमਖ਼Λ͓ئ͍Ͱ͖·͢ʁʯ
    PMʮঝ஌͠·ͨ͠ɻদా͞Μ͓ئ͍͠·͢ɻʯ
    দాʮ͸͍ɻʯ
    

    View Slide

38. ಺੡࣌୅
    ͋Εʁ
    ʮ͜ͷલ΋ґཔ͞ΕͨΑ͏ͳʯ
    ʮ࠷ۙɺ։ൃͰ͖͍ͯͳ͍ʯ
    ʮਓ͕଍Γͳ͍ʯ
    

    View Slide

39. ಺੡࣌୅
    τΠϧʹ૶ૡ͖ۤ͠Ή࣌୅ͷ౸དྷ
    

    View Slide

40. τΠϧ๾໓
    τΠϧ๾໓ͷաఔʹ͍ͭͯ঺հ
    photo by Yamato Fukui
    

    View Slide

41. ํ਑
    Twelve-Factor AppΛ
    Managed ServiceΛ
    CI / CD
    

    View Slide

42. ํ਑
    

    View Slide

43. Twelve-Factor App
    ࣌ؒɾίετͷ࠷খԽ
    Ҡ২ੑ޲্
    Πϯϑϥ؅ཧ͔Βͷ։์
    ܧଓతσϓϩΠ
    εέʔϧΞοϓ
    IUUQTGBDUPSOFU
    

    View Slide

44. Managed Service
    Πϯϑϥ؅ཧෆཁ
    ӡ༻ϦεΫ௿ݮ
    ܧଓతվળ
    

    View Slide

45. CI / CD
    ख࡞ۀ͔Βͷ։์
    ඼࣭޲্
    

    View Slide

46. τΠϧ๾໓
    Case 1
    

    View Slide

47. τΠϧ๾໓લͷঢ়گ
    ؀ڥߏங࡞ۀ
    ৽͍͠ΤϯδχΞ͕ࢀը
    README௨Γʹ؀ڥߏங
    ಈ͔ͳ͍
    ৬ਓͷग़ਞ
    

    View Slide

48. ػೳ։ൃ։࢝·Ͱʹ͔͔࣌ؒΔ
    ৬ਓͷՔಇ͕औΒΕΔ
    τΠϧ๾໓લͷঢ়گ
    ՝୊
    

    View Slide

49. τΠϧ๾໓ઓུ
    Docker Compose
    

    View Slide

50. τΠϧ๾໓ઓུ
    ίϯςφܕԾ૝؀ڥ
    ؀ڥࠩҟ͕খ͘͞ͳΔ
    Ҡ২ੑ͕ߴ͍
    Docker
    

    View Slide

51. τΠϧ๾໓ઓུ
    Docker Compose
    ෳ਺ίϯςφΛఆٛ
    ෳ਺ίϯςφΛ࣮ߦ
    docker-compose up ͷΈ
    

    View Slide

52. ͙͢։ൃͰ͖ΔΑ͏ʹͳͬͨ
    ৬ਓͷग़ਞ͕཈͑ΒΕͨ
    ؀ڥࠩҟʹΑΔόά͕཈͑ΒΕͨ
    τΠϧ๾໓ޙ
    վળޮՌ
    

    View Slide

53. τΠϧ๾໓ޙ
    

    View Slide

54. τΠϧ๾໓
    Case 2
    

    View Slide

55. खॱॻͷ࡞੒
    1୆ͣͭॖୀɾߋ৽
    νΣοΧʔͱෳ਺ਓମ੍
    σϓϩΠ࡞ۀ
    τΠϧ๾໓લͷঢ়گ
    

    View Slide

56. σϓϩΠखॱ൥ࡶʹΑΔ
    ώϡʔϚϯΤϥʔϦεΫ
    σϓϩΠʹର͢Δ޻਺͕ߴΊ
    τΠϧ๾໓લͷঢ়گ
    ՝୊
    

    View Slide

57. τΠϧ๾໓ઓུ
    GitLab CI
    Amazon ECS
    

    View Slide

58. τΠϧ๾໓ઓུ
    ίϯςφΦʔέετϨʔγϣϯ
    αʔό؅ཧෆཁ
    εέδϡʔϧ࣮ߦ
    Amazon ECS
    

    View Slide

59. τΠϧ๾໓ઓུ
    GitϦϙδτϦ
    CI / CDػೳ
    issue؅ཧ
    ίʔυϨϏϡʔ
    GitLab
    

    View Slide

60. RDS
    ELB
    X-Ray
    ECS
    App Server Service
    X-Ray
    Batch Task
    ELB
    ELB
    ɾɾɾ
    X-Ray
    X-Ray
    Coupon Server Service
    ɾɾɾ
    X-Ray
    X-Ray
    Admin Server Service
    ɾɾɾ
    X-Ray
    CloudWatch
    Logs
    S3
    τΠϧ๾໓ઓུ
    

    View Slide

61. ιʔεमਖ਼
    ϨϏϡʔ
    ϙνΔ
    τΠϧ๾໓ޙ
    σϓϩΠखॱ
    

    View Slide

62. ਓҝతϛε͕ൃੜ͠ͳ͘ͳͬͨ
    σϓϩΠʹର͢Δ޻਺͕ݮͬͨ
    վળޮՌ
    τΠϧ๾໓ޙ
    

    View Slide

63. /hoge/ͰPOST࣌͸IP੍ݶ͍ͨ͠
    τΠϧ๾໓ޙ
    

    View Slide

64. τΠϧ๾໓ޙ
    SQLΫΤϦվળ
    

    View Slide

65. ECR
    X-Ray
    APP Server Service
    X-Ray
    New Task
    Old Task
    1. Image Build & Push New Image
    2. Create New Task Definition
    3. Update Service
    1. Push Source Code
    X-Ray
    App Task: 1
    X-Ray
    App Task: 2
    GitLab
    2. Create Merge Request(MR)
    3. Review & LGTM & Accept MR
    τΠϧ๾໓ޙ
    

    View Slide

66. τΠϧ๾໓
    ͓·͚
    

    View Slide

67. τΠϧ๾໓લͷঢ়گ
    ੩తϑΝΠϧσϓϩΠ
    ιʔεमਖ਼
    ϨϏϡʔ
    ϙνΔ
    

    View Slide

68. ɻΛ෇༩͍ͨ͠
    τΠϧ๾໓ޙ
    

    View Slide

69. 1. npm install & build
    2. Put Artifacts to S3 Bucket
    1. Push Source Code
    GitLab
    2. Create Merge Request(MR)
    3. Review & LGTM & Accept MR
    AWS CLI
    τΠϧ๾໓ޙ
    

    View Slide

70. τΠϧ๾໓
    ߋʹ͓·͚
    

    View Slide

71. Ϣχοτςετ
    ίʔσΟϯάن໿֬ೝ
    ෳࡶ౓֬ೝ
    τΠϧ๾໓લͷঢ়گ
    ιʔείʔυϨϏϡ࡞ۀ
    

    View Slide

72. 1. Unit Test
    2. Flake8
    3. Radon
    GitLab
    ςετ࣮ࢪʹΑΔ඼࣭୲อ
    ίʔσΟϯάن໿ͷ֬ೝ
    ίʔυͷෳࡶ౓Λ֬ೝ
    1. Push Source Code
    2. Create Merge Request(MR)
    3. Review & LGTM & Accept MR
    τΠϧ๾໓ޙ
    

    View Slide

73. τΠϧ๾໓ޙ
    

    View Slide

74. τΠϧ๾໓
    ͓·͚
    (͜ΕͰ࠷ޙ)
    

    View Slide

75. τΠϧ๾໓લͷঢ়گ
    όοναʔόෆཁ
    ECSͰఆظతʹ࣮ߦͰ͖Δ
    ࢖ͬͨ෼͚ͩͷྉۚ
    

    View Slide

76. τΠϧ๾໓ޙ
    

    View Slide

77. τΠϧ๾໓
    Case 3
    

    View Slide

78. τΠϧ๾໓લͷঢ়گ
    ֤αʔό͔ΒΞΫηεϩάऔಘ
    grepɾawk৬ਓ
    ಛఆϢʔβͷΞΫηεௐࠪ
    

    View Slide

79. ௐࠪʹ͕͔͔࣌ؒΔ
    ώϡʔϚϯΤϥʔʹΑΔϦεΫ
    τΠϧ๾໓લͷঢ়گ
    ՝୊
    

    View Slide

80. τΠϧ๾໓ઓུ
    CloudWatch Logs
    

    View Slide

81. τΠϧ๾໓ઓུ
    ϩάͷ؂ࢹ
    ϩάͷอଘ
    ϩά΁ͷΞΫηε
    CloudWatch Logs
    

    View Slide

82. τΠϧ๾໓ઓུ
    settings.py
    

    View Slide

83. τΠϧ๾໓ઓུ
    docker-compose.yml
    

    View Slide

84. ਺෼Ͱௐࠪ׬ྃ
    ਓҝతϛεʹΑΔαʔϏεఀࢭ͕ͳ͍
    ඇΤϯδχΞͰ΋ௐࠪՄೳ
    τΠϧ๾໓
    վળޮՌ
    

    View Slide

85. τΠϧ๾໓ޙ
    ϩάΛҰՕॴʹू໿
    લํҰகݕࡧ
    ೔࣌ݕࡧ
    

    View Slide

86. τΠϧ๾໓ޙ
    

    View Slide

87. τΠϧ๾໓
    Case 4
    

    View Slide

88. τΠϧ๾໓લͷঢ়گ
    RDSεέʔϧΞοϓ࡞ۀ
    ࣮ߦ͞ΕΔSQL͕ѱ͍
    CPU࢖༻཰100%ுΓ෇͖
    αʔϏεμ΢ϯൃੜ
    ۚͰղܾ
    

    View Slide

89. αʔϏεμ΢ϯͷՄೳੑ
    αʔϏεҡ࣋ʹίετ͕͔͔Δ
    ࣮૷༧ఆͩͬͨ΋ͷ͕஗Ԇ
    τΠϧ๾໓લͷঢ়گ
    ՝୊
    

    View Slide

90. τΠϧ๾໓ઓུ
    Django QuerySet
    

    View Slide

91. σʔλϕʔεͷ҆ఆՔಇ
    ༨ܭͳεέʔϧΞοϓΛ཈͑Δ
    վળޮՌ
    τΠϧ๾໓ޙ
    

    View Slide

92. ήʔϜ࣮ࢪՄ൱API
    ήʔϜ͸1೔1ճ·Ͱ
    ΞϓϦશମͰ1೔ʹ10,000ճ
    τΠϧ๾໓ޙ
    

    View Slide

93. τΠϧ๾໓ޙ
    

    View Slide

94. τΠϧ๾໓ޙ
    

    View Slide

95. PushͰήʔϜར༻ऀ૿Ճ
    ϨεϙϯεྼԽ
    ΠϯελϯεεέʔϧΞοϓ
    τΠϧ๾໓ޙ
    

    View Slide

96. τΠϧ๾໓ޙ
    

    View Slide

97. τΠϧ๾໓ޙ
    

    View Slide

98. τΠϧ๾໓ޙ
    

    View Slide

99. τΠϧ๾໓
    Case 5
    

    View Slide

100. τΠϧ๾໓લͷঢ়گ
     ؅ཧαʔόʹϩάΠϯͰ͖ͳ͍໰୊ͷௐࠪ
     ϩάௐࠪ
     ϩάʹు͔Ε͍ͯͳ͍
     ιʔεΛݟΔ
     

     View Slide

101. ໰୊ղܾʹ͕͔͔࣌ؒΔ
     τΠϧ๾໓લͷঢ়گ
     ՝୊
     

     View Slide

102. τΠϧ๾໓લͷঢ়گ
     

     View Slide

103. τΠϧ๾໓લͷঢ়گ
     Effective Python

     Item 14: Prefer Exceptions to returning None
     νϟϦ
     Effective
     Python
     —
     

     View Slide

104. τΠϧ๾໓લͷঢ়گ
     ྑ͘ͳ͍఺
     except۟ͰϩΪϯά͍ͯ͠ͳ͍
     except۟ͰNoneΛฦ٫
     

     View Slide

105. τΠϧ๾໓ઓུ
     Django Logging
     

     View Slide

106. ϩάΛݟΔ͚ͩͰঢ়گ೺Ѳ
     τΠϧ๾໓
     վળޮՌ
     

     View Slide

107. τΠϧ๾໓ޙ
     

     View Slide

108. ྑ͘ͳͬͨ఺
     ϩάͰঢ়گΛ೺ѲͰ͖Δ
     ಛผͳྫ֎Λ౤͛Δ͜ͱͰ
     ΍Γ͍ͨ͜ͱΛ໌֬ʹͨ͠
     τΠϧ๾໓ޙ
     

     View Slide

109. τΠϧ๾໓
     Case 6
     

     View Slide

110. τΠϧ๾໓લͷঢ়گ
     ֤APIͷϨΠςϯγௐࠪ
     ֤αʔό͔ΒΞΫηεϩάऔಘ
     grepɾawk৬ਓ
     

     View Slide

111. ͕͔͔࣌ؒΔ
     ϦΞϧλΠϜʹΘ͔Βͳ͍
     τΠϧ๾໓લͷঢ়گ
     ՝୊
     

     View Slide

112. τΠϧ๾໓ઓུ
     AWS X-Ray
     

     View Slide

113. τΠϧ๾໓ઓུ
     ϦΫΤετʹؔ͢Δσʔλऩू
     σʔλͷදࣔɾϑΟϧλϦϯά
     தؒͷαʔϏεৄࡉ΋දࣔ
     AWS X-Ray
     

     View Slide

114. τΠϧ๾໓ઓུ
     X-Ray
     X-Ray
     Daemon
     

     View Slide

115. τΠϧ๾໓ઓུ
     settings.py
     

     View Slide

116. τΠϧ๾໓ઓུ
     Dockerfile
     

     View Slide

117. ຊ൪؀ڥͷঢ়گ͕೺ѲՄೳ
     ύϑΥʔϚϯενϡʔχϯά˕
     ো֐ରԠͰ໾ʹཱͭ
     վળޮՌ
     τΠϧ๾໓ޙ
     

     View Slide

118. τΠϧ๾໓ޙ
     

     View Slide

119. τΠϧ๾໓ޙ
     

     View Slide

120. τΠϧ๾໓ޙ
     

     View Slide

121. τΠϧ๾໓ޙ
     

     View Slide

122. ޮՌ
     τΠϧ๾໓ʹΑΔޮՌͷ঺հ
     photo by Yamato Fukui
     

     View Slide

123. ޮՌ
     ো֐ରԠ͕ݮͬͨ
     ख࡞ۀݮʹ൐͏ώϡʔϚϯΤϥʔ͕গͳ͘ͳͬͨ͜ͱ͕େ͖͍
     ΤϯδχΞϦϯάʹׂ͚Δ͕࣌ؒ૿͑ͨ
     DjangoΞϓϦέʔγϣϯ։ൃʹ஫ྗͰ͖ΔΑ͏ʹͳͬͨ
     ΩϟϦΞΞοϓͰ͖ͨ
     ϦʔμʹͳΓ·ͨ͠
     

     View Slide

124. ·ͱΊ
     photo by Yamato Fukui
     

     View Slide

125. ·ͱΊ
     ʮͦΕτΠϧ͡ΌͶʁʯͷਫ਼ਆ
     τΠϧΛݮΒͤ͹ΤϯδχΞϦϯάʹूதͰ͖Δ
     ܧଓͯ͠վળͰ͖Δঢ়ଶΛ࡞Δ
     τΠϧ͸ແݶʹ༙͖ଓ͚Δ
     Managed ServiceɾCI / CDͷ׆༻
     ؅ཧର৅ΛݮΒ͠ɺࣗಈԽΛਪਐ͢Δ
     

     View Slide

126. ·ͱΊ
     τΠϧΛ๾໓ͯ͠
     PythonͰΠϊϕʔςΟϒͳ
     ࢓ࣄΛ͠Α͏ʂ
     

     View Slide

127. ࢀߟจݙ
     O'Reilly Japan - SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά
     O'Reilly Japan - Effective Python
     

     View Slide