Djangoアプリケーションにおけるトイル撲滅戦記.pdf

by Yasuaki Matsuda

Slide 1

Slide 1 text

DjangoΞϓϦέʔγϣϯʹ͓͚Δ τΠϧ๾໓ઓه

Slide 2

Slide 2 text

ࣗݾ঺հ দా ߁ষ (Yasuaki Matsuda) @yacchin0101 גࣜձࣾΞΠϦοδ αʔόαΠυΤϯδχΞ

Slide 3

Slide 3 text

ࣗݾ঺հ

Slide 4

Slide 4 text

ͲΜͳਓ͕ର৅ʁ τΠϧ๾໓ʹڵຯ͕͋Δ ݱ৔ͰDjangoΛ࢖ͬͯΔ ೔ʑͷӡ༻ۀ຿Ͱർฐ͍ͯ͠Δਓ

Slide 5

Slide 5 text

໨࣍ લ࢙ τΠϧ๾໓ τΠϧͱ͸ ޮՌ ·ͱΊ

Slide 6

Slide 6 text

େมਃ͠༁͍͟͝·ͤΜ τΠϧΛ๾໓ͨ͠Θ͚Ͱ͸ͳ͍

Slide 7

Slide 7 text

େมਃ͠༁͍͟͝·ͤΜ SRE͸ʮτΠϧͷ๾໓ʯ͕ϝΠϯ

Slide 8

Slide 8 text

τΠϧͱ͸ τΠϧʹ͍ͭͯͷ঺հ photo by Yamato Fukui

Slide 9

Slide 9 text

SREͱ͸ Google͕ࣾఏএɾ࣮ફ͍ͯ͠Δ γεςϜ؅ཧͱαʔϏεӡ༻ͷํ๏࿦

Slide 10

Slide 10 text

SREͱ͸ IUUQTMBOEJOHHPPHMFDPNTSFCPPLJOEFYIUNM 5ষ τΠϧͷ๾໓ τΧήʁ SRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά

Slide 11

Slide 11 text

τΠϧͱ͸ ຊ൪αʔϏεʹؔ͢Δ࡞ۀͰ ख࡞ۀͰ܁Γฦ͠ߦΘΕ ࣗಈԽ͢Δ͜ͱ͕ՄೳͰ͋Γ ઓज़తͰ௕ظతͳՁ஋Λ࣋ͨͣ ࡞ۀྔ͕αʔϏεͷ੒௕ʹൺྫ͢Δ

Slide 12

Slide 12 text

৯ثͷखચ͍ ਎ۙͳτΠϧ

Slide 13

Slide 13 text

਎ۙͳτΠϧ ख࡞ۀͰ͋Δ ܁Γฦ͞ΕΔ ࣗಈԽͰ͖Δ ઓज़తͰͳ͍ ௕ظతͳՁ஋Λ࣋ͨͳ͍ αʔϏε੒௕ʹରͯ͠O(n)Ͱͳ͍ ৯ثͷखચ͍

Slide 14

Slide 14 text

਎ۙͳτΠϧ શࣗಈ৯ચػ

Slide 15

Slide 15 text

τΠϧ͕΋ͨΒ͢΋ͷ ҰॠͰՔಇΛຒΊਚ͘͞ΕΔ ϔΠτΛు͘Α͏ʹͳΔ ΩϟϦΞ͕ఀ଺͢Δ ͳͲ

Slide 16

Slide 16 text

·ͱΊ τΠϧΛ๾໓ͯ͠ PythonͰΠϊϕʔςΟϒͳ ࢓ࣄΛ͠Α͏ʂ

Slide 17

Slide 17 text

લ࢙ τΠϧ๾໓લͷDjangoΞϓϦέʔγϣϯʹ͍ͭͯͷ঺հ photo by Yamato Fukui

Slide 18

Slide 18 text

લ࢙ Online to Offline

Slide 19

Slide 19 text

લ࢙ དྷళൢଅࢪࡦ Push௨஌ ϙΠϯτ Ϋʔϙϯ ήʔϜ

Slide 20

Slide 20 text

Push௨஌ ͓ɺ෰ങ͍ʹ ߦ͜͏͔ͳɻ ΫϦΞϥϯεηʔϧ ։࠵தʂ

Slide 21

Slide 21 text

ϙΠϯτ ϙΠϯτ໯͑Δ͠ ߦ͜͏͔ͳɻ དྷళϙΠϯτ Ωϟϯϖʔϯ࣮ࢪதʂ

Slide 22

Slide 22 text

Ϋʔϙϯ ͋ͷళͷΫʔϙϯ ͋Δ͡ΌΜɻ 10%OFFΫʔϙϯ ഑෍தʂ

Slide 23

Slide 23 text

ήʔϜ ΍ͬͨͥ ͓ΊͰͱ͏ʂ 100PTήοτʂ

Slide 24

Slide 24 text

લ࢙ hogeΞϓϦ Push௨஌ ϙΠϯτ Ϋʔϙϯ ήʔϜ ܾࡁ

Slide 25

Slide 25 text

Offline ͓ళͷηʔϧ৘ใΛݟΔ ϙΠϯτ࢒ߴΛ֬ೝ͢Δ ήʔϜΛ࣮ࢪͯ͠ϙΠϯτ֫ಘ

Slide 26

Slide 26 text

Online ͓ళͰΫʔϙϯΛ࢖͏ ΞϓϦͰܾࡁ͢Δ

Slide 27

Slide 27 text

લ࢙ hogeΞϓϦͱࢲͷؔ܎ αʔόαΠυػೳ࣮૷ Πϯϑϥߏங ӡ༻ɾอक୲౰ऀ

Slide 28

Slide 28 text

લ࢙ CPU࢖༻཰ Push௨஌

Slide 29

Slide 29 text

લ࢙ ֎஫࣌୅ ಺੡࣌୅ τΠϧ๾໓࣌୅ 2015೥ 2016೥ 2018೥

Slide 30

Slide 30 text

֎஫࣌୅ ελʔτΞοϓاۀ ։ൃऀෆ଍ ֎஫

Slide 31

Slide 31 text

ʢ2015.x.xʣ ߏ੒ App & Batch Server Admin Server Coupon Server ELB ELB

Slide 32

Slide 32 text

֎஫࣌୅ ຯΘ͍ਂ͍ߏ੒ DBಉډ໰୊ SSLূ໌ॻ௚ஔ͖໰୊ σϓϩΠ: SCP ͳͲ

Slide 33

Slide 33 text

ઌํʮΞϓϦͷػೳ֦ு͍ͨ͠ʯ ֎஫࣌୅ Ӧۀʮঝ஌͍ͨ͠·ͨ͠ʂʯ ೖࣾͨ͠দాʮΑΖ͓͘͠Ͷ͕͍͠·͢ʂʯ PMʮࠓޙͷ֦ுΛߟ͑ͯ಺੡͢Δ͔ʯ

Slide 34

Slide 34 text

಺੡࣌୅ ࣾ಺։ൃ Python / Django Ͱ։ൃ GitLabͰιʔε؅ཧ AWSͰߏங

Slide 35

Slide 35 text

ʢ2016.x.xʣ ߏ੒ App Server Admin Server Coupon Server RDS ELB ELB Batch Server

Slide 36

Slide 36 text

಺੡࣌୅ ଟগ͸վળ͞Εͨ RDS(ϚωʔδυαʔϏε) PHP͔ΒDjango΁Ҡߦ σϓϩΠ: Git Pull ͳͲ

Slide 37

Slide 37 text

ϩʔϯνޙ ઌํʮ͜͜ͷจݴमਖ਼Λ͓ئ͍Ͱ͖·͢ʁʯ PMʮঝ஌͠·ͨ͠ɻদా͞Μ͓ئ͍͠·͢ɻʯ দాʮ͸͍ɻʯ

Slide 38

Slide 38 text

಺੡࣌୅ ͋Εʁ ʮ͜ͷલ΋ґཔ͞ΕͨΑ͏ͳʯ ʮ࠷ۙɺ։ൃͰ͖͍ͯͳ͍ʯ ʮਓ͕଍Γͳ͍ʯ

Slide 39

Slide 39 text

಺੡࣌୅ τΠϧʹ૶ૡ͖ۤ͠Ή࣌୅ͷ౸དྷ

Slide 40

Slide 40 text

τΠϧ๾໓ τΠϧ๾໓ͷաఔʹ͍ͭͯ঺հ photo by Yamato Fukui

Slide 41

Slide 41 text

ํ਑ Twelve-Factor AppΛ Managed ServiceΛ CI / CD

Slide 42

Slide 42 text

ํ਑

Slide 43

Slide 43 text

Twelve-Factor App ࣌ؒɾίετͷ࠷খԽ Ҡ২ੑ޲্ Πϯϑϥ؅ཧ͔Βͷ։์ ܧଓతσϓϩΠ εέʔϧΞοϓ IUUQTGBDUPSOFU

Slide 44

Slide 44 text

Managed Service Πϯϑϥ؅ཧෆཁ ӡ༻ϦεΫ௿ݮ ܧଓతվળ

Slide 45

Slide 45 text

CI / CD ख࡞ۀ͔Βͷ։์ ඼࣭޲্

Slide 46

Slide 46 text

τΠϧ๾໓ Case 1

Slide 47

Slide 47 text

τΠϧ๾໓લͷঢ়گ ؀ڥߏங࡞ۀ ৽͍͠ΤϯδχΞ͕ࢀը README௨Γʹ؀ڥߏங ಈ͔ͳ͍ ৬ਓͷग़ਞ

Slide 48

Slide 48 text

ػೳ։ൃ։࢝·Ͱʹ͔͔࣌ؒΔ ৬ਓͷՔಇ͕औΒΕΔ τΠϧ๾໓લͷঢ়گ ՝୊

Slide 49

Slide 49 text

τΠϧ๾໓ઓུ Docker Compose

Slide 50

Slide 50 text

τΠϧ๾໓ઓུ ίϯςφܕԾ૝؀ڥ ؀ڥࠩҟ͕খ͘͞ͳΔ Ҡ২ੑ͕ߴ͍ Docker

Slide 51

Slide 51 text

τΠϧ๾໓ઓུ Docker Compose ෳ਺ίϯςφΛఆٛ ෳ਺ίϯςφΛ࣮ߦ docker-compose up ͷΈ

Slide 52

Slide 52 text

͙͢։ൃͰ͖ΔΑ͏ʹͳͬͨ ৬ਓͷग़ਞ͕཈͑ΒΕͨ ؀ڥࠩҟʹΑΔόά͕཈͑ΒΕͨ τΠϧ๾໓ޙ վળޮՌ

Slide 53

Slide 53 text

τΠϧ๾໓ޙ

Slide 54

Slide 54 text

τΠϧ๾໓ Case 2

Slide 55

Slide 55 text

खॱॻͷ࡞੒ 1୆ͣͭॖୀɾߋ৽ νΣοΧʔͱෳ਺ਓମ੍ σϓϩΠ࡞ۀ τΠϧ๾໓લͷঢ়گ

Slide 56

Slide 56 text

σϓϩΠखॱ൥ࡶʹΑΔ ώϡʔϚϯΤϥʔϦεΫ σϓϩΠʹର͢Δ޻਺͕ߴΊ τΠϧ๾໓લͷঢ়گ ՝୊

Slide 57

Slide 57 text

τΠϧ๾໓ઓུ GitLab CI Amazon ECS

Slide 58

Slide 58 text

τΠϧ๾໓ઓུ ίϯςφΦʔέετϨʔγϣϯ αʔό؅ཧෆཁ εέδϡʔϧ࣮ߦ Amazon ECS

Slide 59

Slide 59 text

τΠϧ๾໓ઓུ GitϦϙδτϦ CI / CDػೳ issue؅ཧ ίʔυϨϏϡʔ GitLab

Slide 60

Slide 60 text

RDS ELB X-Ray ECS App Server Service X-Ray Batch Task ELB ELB ɾɾɾ X-Ray X-Ray Coupon Server Service ɾɾɾ X-Ray X-Ray Admin Server Service ɾɾɾ X-Ray CloudWatch Logs S3 τΠϧ๾໓ઓུ

Slide 61

Slide 61 text

ιʔεमਖ਼ ϨϏϡʔ ϙνΔ τΠϧ๾໓ޙ σϓϩΠखॱ

Slide 62

Slide 62 text

ਓҝతϛε͕ൃੜ͠ͳ͘ͳͬͨ σϓϩΠʹର͢Δ޻਺͕ݮͬͨ վળޮՌ τΠϧ๾໓ޙ

Slide 63

Slide 63 text

/hoge/ͰPOST࣌͸IP੍ݶ͍ͨ͠ τΠϧ๾໓ޙ

Slide 64

Slide 64 text

τΠϧ๾໓ޙ SQLΫΤϦվળ

Slide 65

Slide 65 text

ECR X-Ray APP Server Service X-Ray New Task Old Task 1. Image Build & Push New Image 2. Create New Task Deﬁnition 3. Update Service 1. Push Source Code X-Ray App Task: 1 X-Ray App Task: 2 GitLab 2. Create Merge Request(MR) 3. Review & LGTM & Accept MR τΠϧ๾໓ޙ

Slide 66

Slide 66 text

τΠϧ๾໓ ͓·͚

Slide 67

Slide 67 text

τΠϧ๾໓લͷঢ়گ ੩తϑΝΠϧσϓϩΠ ιʔεमਖ਼ ϨϏϡʔ ϙνΔ

Slide 68

Slide 68 text

ɻΛ෇༩͍ͨ͠ τΠϧ๾໓ޙ

Slide 69

Slide 69 text

1. npm install & build 2. Put Artifacts to S3 Bucket 1. Push Source Code GitLab 2. Create Merge Request(MR) 3. Review & LGTM & Accept MR AWS CLI τΠϧ๾໓ޙ

Slide 70

Slide 70 text

τΠϧ๾໓ ߋʹ͓·͚

Slide 71

Slide 71 text

Ϣχοτςετ ίʔσΟϯάن໿֬ೝ ෳࡶ౓֬ೝ τΠϧ๾໓લͷঢ়گ ιʔείʔυϨϏϡ࡞ۀ

Slide 72

Slide 72 text

1. Unit Test 2. Flake8 3. Radon GitLab ςετ࣮ࢪʹΑΔ඼࣭୲อ ίʔσΟϯάن໿ͷ֬ೝ ίʔυͷෳࡶ౓Λ֬ೝ 1. Push Source Code 2. Create Merge Request(MR) 3. Review & LGTM & Accept MR τΠϧ๾໓ޙ

Slide 73

Slide 73 text

τΠϧ๾໓ޙ

Slide 74

Slide 74 text

τΠϧ๾໓ ͓·͚ (͜ΕͰ࠷ޙ)

Slide 75

Slide 75 text

τΠϧ๾໓લͷঢ়گ όοναʔόෆཁ ECSͰఆظతʹ࣮ߦͰ͖Δ ࢖ͬͨ෼͚ͩͷྉۚ

Slide 76

Slide 76 text

τΠϧ๾໓ޙ

Slide 77

Slide 77 text

τΠϧ๾໓ Case 3

Slide 78

Slide 78 text

τΠϧ๾໓લͷঢ়گ ֤αʔό͔ΒΞΫηεϩάऔಘ grepɾawk৬ਓ ಛఆϢʔβͷΞΫηεௐࠪ

Slide 79

Slide 79 text

ௐࠪʹ͕͔͔࣌ؒΔ ώϡʔϚϯΤϥʔʹΑΔϦεΫ τΠϧ๾໓લͷঢ়گ ՝୊

Slide 80

Slide 80 text

τΠϧ๾໓ઓུ CloudWatch Logs

Slide 81

Slide 81 text

τΠϧ๾໓ઓུ ϩάͷ؂ࢹ ϩάͷอଘ ϩά΁ͷΞΫηε CloudWatch Logs

Slide 82

Slide 82 text

τΠϧ๾໓ઓུ settings.py

Slide 83

Slide 83 text

τΠϧ๾໓ઓུ docker-compose.yml

Slide 84

Slide 84 text

਺෼Ͱௐࠪ׬ྃ ਓҝతϛεʹΑΔαʔϏεఀࢭ͕ͳ͍ ඇΤϯδχΞͰ΋ௐࠪՄೳ τΠϧ๾໓ վળޮՌ

Slide 85

Slide 85 text

τΠϧ๾໓ޙ ϩάΛҰՕॴʹू໿ લํҰகݕࡧ ೔࣌ݕࡧ

Slide 86

Slide 86 text

τΠϧ๾໓ޙ

Slide 87

Slide 87 text

τΠϧ๾໓ Case 4

Slide 88

Slide 88 text

τΠϧ๾໓લͷঢ়گ RDSεέʔϧΞοϓ࡞ۀ ࣮ߦ͞ΕΔSQL͕ѱ͍ CPU࢖༻཰100%ுΓ෇͖ αʔϏεμ΢ϯൃੜ ۚͰղܾ

Slide 89

Slide 89 text

αʔϏεμ΢ϯͷՄೳੑ αʔϏεҡ࣋ʹίετ͕͔͔Δ ࣮૷༧ఆͩͬͨ΋ͷ͕஗Ԇ τΠϧ๾໓લͷঢ়گ ՝୊

Slide 90

Slide 90 text

τΠϧ๾໓ઓུ Django QuerySet

Slide 91

Slide 91 text

σʔλϕʔεͷ҆ఆՔಇ ༨ܭͳεέʔϧΞοϓΛ཈͑Δ վળޮՌ τΠϧ๾໓ޙ

Slide 92

Slide 92 text

ήʔϜ࣮ࢪՄ൱API ήʔϜ͸1೔1ճ·Ͱ ΞϓϦશମͰ1೔ʹ10,000ճ τΠϧ๾໓ޙ

Slide 93

Slide 93 text

τΠϧ๾໓ޙ

Slide 94

Slide 94 text

τΠϧ๾໓ޙ

Slide 95

Slide 95 text

PushͰήʔϜར༻ऀ૿Ճ ϨεϙϯεྼԽ ΠϯελϯεεέʔϧΞοϓ τΠϧ๾໓ޙ

Slide 96

Slide 96 text

τΠϧ๾໓ޙ

Slide 97

Slide 97 text

τΠϧ๾໓ޙ

Slide 98

Slide 98 text

τΠϧ๾໓ޙ

Slide 99

Slide 99 text

τΠϧ๾໓ Case 5

Slide 100

Slide 100 text

τΠϧ๾໓લͷঢ়گ ؅ཧαʔόʹϩάΠϯͰ͖ͳ͍໰୊ͷௐࠪ ϩάௐࠪ ϩάʹు͔Ε͍ͯͳ͍ ιʔεΛݟΔ

Slide 101

Slide 101 text

໰୊ղܾʹ͕͔͔࣌ؒΔ τΠϧ๾໓લͷঢ়گ ՝୊

Slide 102

Slide 102 text

τΠϧ๾໓લͷঢ়گ

Slide 103

Slide 103 text

τΠϧ๾໓લͷঢ়گ Effective Python  Item 14: Prefer Exceptions to returning None νϟϦ Effective Python —

Slide 104

Slide 104 text

τΠϧ๾໓લͷঢ়گ ྑ͘ͳ͍఺ except۟ͰϩΪϯά͍ͯ͠ͳ͍ except۟ͰNoneΛฦ٫

Slide 105

Slide 105 text

τΠϧ๾໓ઓུ Django Logging

Slide 106

Slide 106 text

ϩάΛݟΔ͚ͩͰঢ়گ೺Ѳ τΠϧ๾໓ վળޮՌ

Slide 107

Slide 107 text

τΠϧ๾໓ޙ

Slide 108

Slide 108 text

ྑ͘ͳͬͨ఺ ϩάͰঢ়گΛ೺ѲͰ͖Δ ಛผͳྫ֎Λ౤͛Δ͜ͱͰ ΍Γ͍ͨ͜ͱΛ໌֬ʹͨ͠ τΠϧ๾໓ޙ

Slide 109

Slide 109 text

τΠϧ๾໓ Case 6

Slide 110

Slide 110 text

τΠϧ๾໓લͷঢ়گ ֤APIͷϨΠςϯγௐࠪ ֤αʔό͔ΒΞΫηεϩάऔಘ grepɾawk৬ਓ

Slide 111

Slide 111 text

͕͔͔࣌ؒΔ ϦΞϧλΠϜʹΘ͔Βͳ͍ τΠϧ๾໓લͷঢ়گ ՝୊

Slide 112

Slide 112 text

τΠϧ๾໓ઓུ AWS X-Ray

Slide 113

Slide 113 text

τΠϧ๾໓ઓུ ϦΫΤετʹؔ͢Δσʔλऩू σʔλͷදࣔɾϑΟϧλϦϯά தؒͷαʔϏεৄࡉ΋දࣔ AWS X-Ray

Slide 114

Slide 114 text

τΠϧ๾໓ઓུ X-Ray X-Ray Daemon

Slide 115

Slide 115 text

τΠϧ๾໓ઓུ settings.py

Slide 116

Slide 116 text

τΠϧ๾໓ઓུ Dockerfile

Slide 117

Slide 117 text

ຊ൪؀ڥͷঢ়گ͕೺ѲՄೳ ύϑΥʔϚϯενϡʔχϯά˕ ো֐ରԠͰ໾ʹཱͭ վળޮՌ τΠϧ๾໓ޙ

Slide 118

Slide 118 text

τΠϧ๾໓ޙ

Slide 119

Slide 119 text

τΠϧ๾໓ޙ

Slide 120

Slide 120 text

τΠϧ๾໓ޙ

Slide 121

Slide 121 text

τΠϧ๾໓ޙ

Slide 122

Slide 122 text

ޮՌ τΠϧ๾໓ʹΑΔޮՌͷ঺հ photo by Yamato Fukui

Slide 123

Slide 123 text

ޮՌ ো֐ରԠ͕ݮͬͨ ख࡞ۀݮʹ൐͏ώϡʔϚϯΤϥʔ͕গͳ͘ͳͬͨ͜ͱ͕େ͖͍ ΤϯδχΞϦϯάʹׂ͚Δ͕࣌ؒ૿͑ͨ DjangoΞϓϦέʔγϣϯ։ൃʹ஫ྗͰ͖ΔΑ͏ʹͳͬͨ ΩϟϦΞΞοϓͰ͖ͨ ϦʔμʹͳΓ·ͨ͠

Slide 124

Slide 124 text

·ͱΊ photo by Yamato Fukui

Slide 125

Slide 125 text

·ͱΊ ʮͦΕτΠϧ͡ΌͶʁʯͷਫ਼ਆ τΠϧΛݮΒͤ͹ΤϯδχΞϦϯάʹूதͰ͖Δ ܧଓͯ͠վળͰ͖Δঢ়ଶΛ࡞Δ τΠϧ͸ແݶʹ༙͖ଓ͚Δ Managed ServiceɾCI / CDͷ׆༻ ؅ཧର৅ΛݮΒ͠ɺࣗಈԽΛਪਐ͢Δ

Slide 126

Slide 126 text

·ͱΊ τΠϧΛ๾໓ͯ͠ PythonͰΠϊϕʔςΟϒͳ ࢓ࣄΛ͠Α͏ʂ

Slide 127

Slide 127 text

ࢀߟจݙ O'Reilly Japan - SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά O'Reilly Japan - Effective Python