Slide 1

Slide 1 text

εΩϚ࣌ؒͰ΋ίʔυ͕ॻ͖͍ͨ ࣄۀ։ൃ෦ɹ੡඼։ൃνʔϜ ొஃऀԣࢁจਓ 

Slide 2

Slide 2 text

 ࣗݾ঺հ

Slide 3

Slide 3 text

ࣗݾ঺հ  wԣࢁจਓ !UVUVUFO  wࣄۀ։ൃ෦੡඼։ൃνʔϜ w&$ɺ$3.޲͚ͷ
 CBDLFOENJDSPTFSWJDFT w+"7"ϓϩάϥϚ w޷͖ͳ৯΂෺͸αʔϞϯʂ w࠷ۙڕհྨͷෆړͷχϡʔε͕ʜ

Slide 4

Slide 4 text

 ࠷ۙίʔυॻ͍ͯΔํʁ

Slide 5

Slide 5 text

 ίʔυΛॻ͘ͷ͕͓࢓ࣄͷํʁ

Slide 6

Slide 6 text

 ࠷ۙϓϩμΫτίʔυॻ͍ͯΔํʁ

Slide 7

Slide 7 text

࠷ۙͷ͓࢓ࣄ  wաڈ࡞੒ͨ͠αʔϏεͷҾ͖ܧ͗ࢿྉ࡞੒ wϚΠΫϩαʔϏεຊ൪Ξοϓσʔτ࡞ۀखॱॻ࡞੒ wϚΠΫϩαʔϏεʹରͯ͠ͷෛՙࢼݧ wϦϦʔε࡞ۀʢ࣋ͪճΓ੍ʣ wۃكʹCVHpY

Slide 8

Slide 8 text

 จॻΛͨͨ͠ΊΔ͓࢓ࣄଟΊ +BWBΛॻ͓͘࢓ࣄ͕΄΅ͳ͠

Slide 9

Slide 9 text

࠷ۙͷ͓࢓ࣄ  wաڈ࡞੒ͨ͠αʔϏεͷҾ͖ܧ͗ࢿྉ࡞੒ wϚΠΫϩαʔϏεຊ൪Ξοϓσʔτ࡞ۀखॱॻ࡞੒ wϚΠΫϩαʔϏεʹରͯ͠ͷෛՙࢼݧ wϦϦʔε࡞ۀʢ࣋ͪճΓ੍ʣ wۃكʹCVHpY

Slide 10

Slide 10 text

 \จॻ +BWB^Λॻ͘࢓ࣄҎ֎

Slide 11

Slide 11 text

ຊൃදͷ͍ͭͯ  wຊൃද͸ొஃऀͷࠓ͋Δٕज़͔Β೧ग़ͨ͠ղ๏ wࠓճ࿩͢಺༰͸ɺʮ࠷దղʯͰ͸ܾͯ͠ແ͍ wʮ΋ͬͱ͍͍ํ๏͕͋ΔΑʂʯͬͯͱ͖͸ϋογϡλάΛͭ ͚ͯπΠʔτ͍ͯͩ͘͠͞ʂʂʢݟ·͢ɺ࣌ʹ͸ฉ͖·͢ʣ

Slide 12

Slide 12 text

঺հࣄྫ  w$BTFσʔλ౤ೖܦաΛݟ͍ͨ w$BTF&$4ͷλεΫͷࢮҼ͕͠Γ͍ͨ w$BTF*TTVFͷ࢒ͬͯΔNJMFTUPOFΛ֬ೝ͍ͨ͠

Slide 13

Slide 13 text

 Case 1 σʔλ౤ೖܦաΛݟ͍ͨ

Slide 14

Slide 14 text

$BTFσʔλ౤ೖܦաΛݟ͍ͨ  w૝ఆͨ݅͠਺Λ࿙Εͳ͘ొ࿥Ͱ͖Δ͔ͷ֬ೝ w౤ೖ·Ͱͷ࣌ؒΛࢀߟ஋ͱͯ͠ܭଌ͍ͨ͠ w%#ͱ&4ͱͷ஗Ԇ͕͋ΔͳΒ͹ଌఆ͍ͨ͠ wσʔλ౤ೖͷεϧʔϓοτ͕౤ೖ࠷தʹམͪͳ͍͔ɺͬ͘͟ Γ֬ೝ͍ͨ͠ wંΕઢάϥϑͰ֬ೝ͍ͨ͠ʂ

Slide 15

Slide 15 text

ߏ੒ਤ 

Slide 16

Slide 16 text

ߏ੒ਤ  DBͱESͷ݅਺͕ Ұக͢Δ͔ͷ֬ೝ

Slide 17

Slide 17 text

ίʔυͰ΍Δ͜ͱɾ΍Βͳ͍͜ͱ  wCBTIͰఆظతʹ%#ͱ&4ΛQPMMJOH͢Δ wඵຖʹऔಘ͍ͨ͠ɻ͔͠͠ɺ݅਺૿͑Δ͝ͱʹ݅਺औಘ࣌ ʹ%#ɼ&4ڞʹ஗Ԇ͕େ͖͘ͳΓɺඵຖ͔ΒͣΕΔɻ
 ΏΔ͘ʮTMFFQʯͰQPMMJOH͢ΔʹͱͲΊ͓ͯ͘ w׬શࣗಈԽΛ໨ࢦ͢Θ͚Ͱ͸ແ͍ͷͰɺ
 άϥϑΛ࡞Δͷ͸ίʔυͰؤுΒͳ͍ wUTWͰϑΝΠϧʹॻ͖ग़ͤ͹(PPHMF4QSFBETIFFUͰάϥϑ Λ࡞Δ͜ͱ͕Ͱ͖Δ

Slide 18

Slide 18 text

EBUFίϚϯυͰ+45औಘ  $ date +"%Y/%m/%d %H:%M:%S %Z" 2019/09/xx 02:44:38 UTC $ TZ=JST-9 date +"%Y/%m/%d %H:%M:%S %Z" 2019/09/xx 11:44:54 JST w&$಺ͷλΠϜκʔϯΛมߋͤͣʹEBUFίϚϯυΛ+45Ͱ
 ֬ೝ͢ΔখٕEBUFίϚϯυͷ಄ʹҎԼΛ͚ͭΔ 5;+45PS5;"TJB5PLZP

Slide 19

Slide 19 text

%#ɺ&4ͷDPVOUΛऔಘ͢Δ  1 # DB 2 DB_COUNT=$(mysql -uroot -p${PASSWD} \ 3 -h${AURORA_ENDPOINT} ${DB_NAME} \ 4 -N -e 'select count(*) from '${TABLE_NAME}) 5 6 # ES 7 ES_COUNT=$(curl -s -H"Content-Type:application/json" \ 8 "http://${ES_ENDPOINT}:9200/${ES_INDEX_NAME}/_search" \ 9 -XPOST -d'{}' | jq '.hits.total') 10

Slide 20

Slide 20 text

ඵ͓͖ʹQPMMJOH͢Δ  1 LOG_NAME=count.log 2 if [ -f ${LOG_NAME} ]; then 3 rm ${LOG_NAME} 4 fi 5 6 while :; do 7 8 # ͜͜ʹDBͱESͷcountΛऔಘ͢ΔॲཧΛೖΕΔ 9 10 printf "%s\t%s\t%s\n" \ 11 "$(TZ=JST-9 date +'%m/%d %H:%M:%S JST')" \ 12 "${DB_COUNT}" "${ES_COUNT}" | tee -a ${LOG_NAME} 13 sleep 5 14 done

Slide 21

Slide 21 text

࣮ߦ݁ՌʢTBNQMFʣ  09/xx 02:09:12 JST 14492481 7606726 09/xx 02:09:21 JST 14498630 7610508 09/xx 02:09:30 JST 14504525 7611873 09/xx 02:09:39 JST 14510488 7615665 09/xx 02:09:49 JST 14516415 7618484 09/xx 02:09:58 JST 14522609 7620923 09/xx 02:10:07 JST 14528732 7624223 09/xx 02:10:16 JST 14535006 7627025 09/xx 02:10:26 JST 14541305 7630306 09/xx 02:10:35 JST 14547499 7633213

Slide 22

Slide 22 text

࣮ߦ݁ՌʢTBNQMFʣ  09/xx 02:09:12 JST 14492481 7606726 09/xx 02:09:21 JST 14498630 7610508 09/xx 02:09:30 JST 14504525 7611873 09/xx 02:09:39 JST 14510488 7615665 09/xx 02:09:49 JST 14516415 7618484 09/xx 02:09:58 JST 14522609 7620923 09/xx 02:10:07 JST 14528732 7624223 09/xx 02:10:16 JST 14535006 7627025 09/xx 02:10:26 JST 14541305 7630306 09/xx 02:10:35 JST 14547499 7633213

Slide 23

Slide 23 text

࣮ߦ݁ՌʢTBNQMFʣ  09/xx 02:09:12 JST 14492481 7606726 09/xx 02:09:21 JST 14498630 7610508 09/xx 02:09:30 JST 14504525 7611873 09/xx 02:09:39 JST 14510488 7615665 09/xx 02:09:49 JST 14516415 7618484 09/xx 02:09:58 JST 14522609 7620923 09/xx 02:10:07 JST 14528732 7624223 09/xx 02:10:16 JST 14535006 7627025 09/xx 02:10:26 JST 14541305 7630306 09/xx 02:10:35 JST 14547499 7633213 σʔλͷಉظ଎౓͕શମతʹ஗Ԇ͕େ͖͘ɺ ऴ൫&4ͷσʔλ਺͕ߥͿ͍ͬͯΔͷͰɺ &4ଆʹͳΜΒ͔ͷݪҼ͕͋Γͦ͏ ͬͯͷ͕ਪଌͰ͖ͨΓ͠·͢

Slide 24

Slide 24 text

࡞ͬͯΈͯ  w&4ͷΠϯελϯελΠϓͰͷੑೳҧ͍Ͱ౤ೖ஗Ԇ໨ʹݟ͑ͯ ௐ੔͠΍͔ͬͨ͢ wදܭࢉΞϓϦͷάϥϑ͸࿑ͤͣݟ΍͍͢ wσʔλྔ͕૿͑ͯདྷΔͱඵຖʹͳΒͳ͍໰୊ wEBUFίϚϯυͷ5;͸໨͔Βྡྷ

Slide 25

Slide 25 text

 Case 2 ECSͷλεΫͷࢮҼ͕஌Γ͍ͨ

Slide 26

Slide 26 text

$BTF&$4ͷλεΫͷࢮҼ͕͠Γ͍ͨ  w&$4Ͱಈ͍ͯΔ&4΁1045͢ΔαʔϏεͷλεΫ͕େྔʹࢮ͵ w໷தʹ௕࣌ؒෛՙݕূ͍ͯ͠ΔͨΊɺλεΫͷ4501৘ใ͕ ".$্͔Βফ͍͑ͯΔ wˢIܦա͢Δͱফ͍͑ͯΔ wʮݕূ͍ͨ݁͠Ռʯ͕ग़ͳ͍ݪҼ͕Θ͔Βͳ͍ͷͰɺௐࠪ͠ͳ ͍Θ͚ʹ͸͍͔ͳ͍ w໷த௕࣌ؒ΋͸Γ͍ͭͯ؂ࢹ΋ͨ͘͠ͳ͍ʢ໷͸༡ͼ͍ͨʣ

Slide 27

Slide 27 text

ߏ੒ਤ  ͜ͷαʔϏεͷλεΫ͕ Α͘མͪΔ

Slide 28

Slide 28 text

λεΫͷ4501৘ใ ݱࡏʣ  ఀࢭݪҼͷ৘ใ͕1࣌ؒܦͭͱ ফ͑ͯɺ௥͑ͳ͘ͳΔ

Slide 29

Slide 29 text

ίʔυͰ΍Δ͜ͱɾ΍Βͳ͍͜ͱ  wఆظ࣮ߦͷબ୒ࢶ͸৭ʑ͋ΔͩΖ͏͚Ͳɺख࣋ͪͷ෢ثͰҰ ൪ѻ͍͍҆"84-BNCEBʢ1ZUIPOʣ "NB[PO $MPVE8BUDI&WFOUT Ҏ߱$8&WFOUT  w&YJUͨ͠λεΫ಺ͷϑΝΠϧ΋ϨετΞͰ͖ͨΒ͋ͱ͋ͱௐ ࠪ͠΍͍͚͢ΕͲ΋ɺख࡞ۀ͔ͭ࢒ͬͯͨΒͷӡ༻Ͱ wϝʔϧ͸ݸਓతʹຒ΋Εͤ͞Δࣗ৴͕͋ΔͷͰɺࣾ಺νϟο τɺࠓճ͸4MBDL΁௨஌ͯ͠ϩάΛ࢒͢ ※ड৴τϨΠͷΠϝʔδը૾Ͱ͢→

Slide 30

Slide 30 text

"84-BNCEB࢖ͬͯ௨஌͠Α͏  w"84$-*Ͱӈͷ৘ใ͕औΕΔ͔֬ೝ wMJTUUBTLT͔Β4UPQͨ͠UBTL"SOΛऔಘ wEFTDSJCFUBTLT͔Β4UPQͨ͠৘ใΛҾ͖ग़ͤΔͷΛ֬ೝ w"84$-*ͰͰ͖Δ͜ͱCPUPͰͰ͖Δ͜ͱ w௨஌͸*ODPNJOH8FCIPPLTΛ࢖͍4MBDL΁ wIͰফ͑ΔͳΒ෼͝ͱ$8&WFOUTͰఆظ࣮ߦ͠ɺલճ࣮ ߦ͠௨஌͓͑ͨ͠UBTL"SOΛ4ʹอଘ

Slide 31

Slide 31 text

"84$-*CPUP  aws ecs list-tasks \ --cluster ${CLUSTER_NAME} \ --service-name ${SERVICE_NAME} \ --desired-status "STOPPED" ecs = boto3.client("ecs") resp = ecs.list_tasks( cluster=CLUSTER_NAME, serviceName=SERVICE_NAME, desiredStatus="STOPPED") wTVCDPNNBOEΛTOBLF DBTFʹม׵ͨ͠ϝιου ໊ʹͳͬͯΔ wύϥϝʔλ໊ΛDBNFM DBTFʹม׵ͨ͠Ҿ਺ʹ ͳ͍ͬͯΔ wSFTQʹ͸KTPO͔ΒQZUIPO ͷEJDU΍MJTU౳ʹม׵͞Ε ͨσʔλ͕ฦͬͯ͘Δ

Slide 32

Slide 32 text

࣮ߦ݁Ռ 

Slide 33

Slide 33 text

࣮ߦ݁Ռ  4MBDLʹϩά͕࢒ΔͷͰɺͲ͏ͯ͠λεΫ͕ࢮΜ͔ͩͲ͏͔͕Θ͔Δɻ ͜ΕΒͷ৘ใΛݩʹ࣍ճҎ߱ͷෛՙࢼݧͰىಈύϥϝʔλ΍λεΫͷϝϞϦྔ౳Λ ௐ੔ͨ͠ΓɺDPOUBJOFS*OTUBODF"SOΑΓDPSFEVNQΛճऩͰ͖ͨΓͨ͠ɻ

Slide 34

Slide 34 text

࡞ͬͯΈͯ  wࢮΜͩλεΫͷௐ͕͍ࠪͩͿָʹͳͬͨ wಋೖલ͸λεΫ͕ࢮΜͩ͜ͱʹؾ͍ͮͨͱ͖ʹ౿Έ୆͔Β &$4ͷϗετΠϯελϯεݸʹରͯ͠ɺ44)ϦϞʔτί ϚϯυΛୟ͍ͯϗετΛ୳ͨ͠ wಛఆͷαʔϏε޲͚ʹ࡞ͬͨͱ͜Ζɺ্ͷਓ͔ΒʮશαʔϏ ε༻ͷཉ͍͠ʯͱ͍͏੠͕Ͱͨ
 ʢ࣌ؒΛݟ͚ͭͯରԠ͍ͨ͠ͳʔʣ

Slide 35

Slide 35 text

 Case 3 Issueͷ࢒ͬͯΔmilestoneΛ֬ೝ͍ͨ͠

Slide 36

Slide 36 text

$BTF*TTVFͷ࢒ͬͯΔNJMFTUPOFΛ֬ೝ͍ͨ͠  wຖिϦϦʔε࡞ۀ͕͋Δ w୲౰͸࣋ͪճΓɺೋਓͰ࡞ۀ͢Δ w࡞ۀͷதʹݸ͋ΔϦϙδτϦͷNJMFTUPOFΛ֬ೝ͠ɺ ࢒ͬͨ*TTVFΛҠಈ͠ɺNJMFTUPOFΛDMPTF͢Δ࡞ۀ͕͋Δ wʮ͑ɺͳʹͦΕ໘౗͍͘͞ɻ͏ͪͷ౰൪͕དྷΔલʹ࢒ͬͯΔ NJMFTUPOF͚ͩݟΕ͹͍͍Α͏ʹ͍ͨ͠ʯ

Slide 37

Slide 37 text

ίʔυͰ΍Δ͜ͱɾ΍Βͳ͍͜ͱ  w࠷ॳ͸CBTIͷGPS PQFOίϚϯυ࢖ͬͯϒϥ΢βʹશ෦දࣔͯ͠໨ HSFQͯ͠୳ͦ͏ˠݸͷλϒ͸ΑΖ͘͠ͳ͍ w"84-BNCEB (JU)VC"1*WΛ࢖͍ɺ֤ϦϙδτϦͷNJMFTUPOF ͱɺNJMFTUPOFʹ࢒ͬͯΔ*TTVF13Λऔಘ wผͷCPUͷίʔυʢOPU࡞ऀʣ͕͋ΔͷͰೝূ෦෼౳ྲྀ༻ w͜ΕΒͷ৘ใΛ4MBDLʹ௨஌ w*ODPNJOH8FCIPPLTͷ63-͸$BTFͷΛྲྀ༻ wNJMFTUPOFʹ࢒͍ͬͯΔ*TTVFͷҠಈɺNJMFTUPOFΛด͡Δॲཧ͸࣮૷͠ ͳ͍ʢʹ֬ೝ͸ϦϦʔε୲౰ऀ͕΍Δʣʢཧ༝͸ޙड़ʣ

Slide 38

Slide 38 text

ߏ੒ਤ  w೔࣌Ͱಈ͔͢ͷͰ$8&WFOUT Ͱ࣮ߦ wखಈͰ࣮ߦ͍ͨ͠ཁ๬ʹԠ͑ TMBTIDPNNBOE "1* (BUFXBZͷ࿈ܞʢޙ೔௥Ճʣ wJDPOΛมߋͰ͖Δ͜ͱΛ͍͍ ͜ͱʹ̍MBNCEBෳ໾ͯ͠Δ

Slide 39

Slide 39 text

࣮ߦ݁Ռ 

Slide 40

Slide 40 text

࣮ߦ݁Ռ  ϦϦʔε೔͸*TTVF13ͷ "TTJOFFͷਓʹϦϓϥΠ͕ͭ͘Α

Slide 41

Slide 41 text

࣮ߦ݁Ռ  ϦϦʔε೔͸NJMFTUPOFͷDMPTF ʹ͍ͭͯ΋֬ೝ͢ΔΑ

Slide 42

Slide 42 text

࡞ͬͯΈͯ  wධ൑͕Α͔ͬͨ͠ɺࣗ෼Ͱ࢖ͬͯΈͯͱͯ΋ศརͩͬͨ wࣗ෼ͷϦϦʔεલ೔ʹཉ͍͠ػೳ͕Ͱ͖ͯͯͦͷ೔ͷ͏ͪʹ࣮૷ͨ͠Γ wࣗಈͰTFNWFS͚ͭΔػೳͱ͔΋ߦ͚ͦ͏ͩΑͶɺͱ͍͏ໝ૝͕͸͔ͲͬͨΓ w·͊ɺ՝୊΋ͨ͘͞Μ w໦༵Ҏ֎ʹϦϦʔε͕ൃੜ͢Δͱ৭ʑͱ่յ͢Δ wॲཧతʹ͸ʮ*TTVFʢ13ʣΛ࣍ͷNJMFTUPOF΁Ҡಈ͢Δʯͱ͍͏ॲཧ΋࣮૷ ͨ͠Γ͢Δ͕ɺۦ͚ࠐΈ*TTVFΛݟಀ͢Մೳੑ͕͋ͬͨͷͰɺॲཧʹ͸૊Έ ࠐΜͰ͍ͳ͍

Slide 43

Slide 43 text

 ۦ͚ࠐΈ͓ئ͍ʢ*TTVFʣ͸ ΍ΊΑ͏ʂ

Slide 44

Slide 44 text

 ·ͱΊ

Slide 45

Slide 45 text

·ͱΊ  wෛՙࢼݧͷதʹ΋ίʔυॻ͚Δͱ͜ΖʢTIFMMɺCPU౳ʣ͕ͨ ͘͞Μ͋Δ wશࣗಈԽ͸ޙʑָ͕͔΋͚ͩͲ΋ɺൃల్্Ͱ΍Γ͗͢Δͱ ӡ༻ʹࢧো͕ͰΔ͔΋ʁʹ΄Ͳ΄Ͳ͕େࣄ wίϐϖͩΖ͏͕ͳΜͩΖ͏ָ͕Λ͢Δ w࡞Εͦ͏ͳΒαΫοͱ࡞ͬͯΈΔ wҊ֎େͨ͠࡞ۀ͡Όͳͯ͘΋ଞͷਓʹ͸ࢗ͞ΔՄೳੑ͕͋Δ

Slide 46

Slide 46 text

No content