Amazon ECSを活用したAWS運用自動化サービスの裏側を包み隠さず解説 / inside opswitch

"NB[PO&$4Λ׆༻ͨ͠ "84ӡ༻ࣗಈԽαʔϏεͷཪଆΛ แΈӅͣ͞ղઆ "84ࣄۀຊ෦ɹϓϩμΫτάϧʔϓ ҏ౻঵!TIPJUP

εϥΠυ͸ޙͰެ։͠·͢ͷͰ ൃදதͷ಺༰ΛϝϞ͢Δඞཁ͸͋Γ·ͤΜɻ ࣸਅࡱӨΛ͢Δ৔߹͸ ϑϥογϡɾγϟολʔԻ͕ग़ͳ͍Α͏ʹ͝഑ྀ͍ͩ͘͞ Attention http://bit.ly/inside-opswitch

ϋογϡλά DNEFWJP DNEFWJP

ηογϣϯ֓ཁΠϕϯταΠτΑΓ࠶ܝ "NB[PO&$4Λ׆༻ͨ͠"84ӡ༻ࣗಈԽαʔϏε ʮPQTXJUDIʯɺͦͷཪଆͰ͸ɺ%PDLFSίϯςφ΍ϫʔ ΫϑϩʔΤϯδϯ"QBDIF"JSqPXɺαʔό؅ཧɾ؂ࢹ αʔϏε.BDLFSFMɺίϥϘϨʔγϣϯϋϒ4MBDLͳͲ ͕׆༻͞Ε͍ͯ·͢ɻ࣮ࡍͷαʔϏεͷΞʔΩςΫ νϟɺ։ൃ؀ڥɺσϓϩΠɺ؂ࢹ·ͰɺแΈӅͣ͞ղઆ ͠·͢ɻ

ηογϣϯͷ໨తئ๬ ঺հ͢Δଟ͘ͷ৘ใͷத͔ΒɺҰͭͰ΋ ։ൃɾӡ༻ͷώϯτʹͳΔ΋ͷΛ͝ఏڙ͢Δ

ࣗݾ঺հ ҏ౻঵!TIPJUP ιϑτ΢ΣΞΤϯδχΞ 4BB4ϓϩμΫτͷاըɾ։ൃɾӡ༻ Ϋϥεϝιουೖ͔ࣾΒ໿೥ܦա ޷͖ͳ"84αʔϏε ⿣ "NB[PO&$4
⿣ "NQMJGZ$POTPMF ⿣ "NB[PO&MBTUJDTFBSDI4FSWJDF

"NB[PO&$4Λ׆༻ͨ͠ "84ӡ༻ࣗಈԽαʔϏεͷཪଆΛ แΈӅͣ͞ղઆ

ΞδΣϯμ ✦ "84ӡ༻ͱ՝୊ ✦ τΠϧͱӡ༻ࣗಈԽ ✦ "84ӡ༻ࣗಈԽαʔϏεͷཪଆ ✦
·ͱΊ

"84ӡ༻ ఆৗ࡞ۀ ো֐ରԠ "84ΠϯϑϥΛ҆શͰ҆ఆͨ͠؀ڥʹҡ࣋ αʔόɺσʔλϕʔεɺωοτϫʔΫɺ"84Ϛωδϝϯτ ίϯιʔϧͳͲΛखॱॻʹԊͬͯૢ࡞ ࣌ؒ೔ͷো֐ରԠΛߦ͍ɺ҆৺ɾ҆શͳ  ؀ڥͷҡ࣋
௨஌΍࠶ىಈͱ͍ͬͨো֐Ұ࣍ରԠ ϩάղੳ΍ݪҼௐࠪɺো֐෮چͱ͍ͬͨೋ࣍ରԠ IUUQTDMBTTNFUIPEKQTFSWJDFTNFNCFSTBXTPQFSBUJOH

ఆৗ࡞ۀͷྫ w&$&#43%4όοΫΞοϓ w&$3%4Πϯελϯεͷى ಈɾఀࢭ w&$ΠϯελϯελΠϓͷม ߋ wηΩϡϦςΟάϧʔϓͷ࡞੒ɾ มߋɾ࡟আ
wηΩϡϦςΟνΣοΫ wύονద༻ w"$.ূ໌ॻͷखಈߋ৽ wυϝΠϯͷखಈߋ৽ w3*ͷߪೖ w&-#1SF8BSNJOHਃ੥ w

ӡ༻ͷࠔΓ͝ͱͷྫ ӡ༻ʹ๩ࡴ͞Εɺকདྷʹ޲͚ͨվળʹ࣌ؒΛׂ͚ͳ͍ wӡ༻୲౰ऀͷ࡞ۀෛՙ͕ߴ͘ɺ͍ͭ΋࢒ۀɺٳ೔ग़ۈ͕ൃੜ wӡ༻υΩϡϝϯτ͕͋ͬͯ΋ϝϯςφϯε͞Ε͍ͯͳ͍ w୲౰ऀʹґଘͨ͠ۀ຿͕͋Γɺͦͷਓʹෛՙ͕ूத wπʔϧ΍ࣗಈԽʹίετ͕͔͚ΒΕͣʹɺզຫͯ͠࢖͍ଓ͚͍ͯΔ wखॱॻʹैͬͯ࡞ۀΛ͢Δ͕Φϖϛε͕ൃੜ͠ɺ࠶ൃ๷ࢭͷνΣοΫʹ࣌ ͕͔͔ؒΔ
wʜ IUUQTDMBTTNFUIPEKQTFSWJDFTNFNCFSTBXTPQFSBUJOH

·ͱΊ

ʮτΠϧʯͷఆٛ ϓϩμΫγϣϯαʔϏεΛಈ࡞ͤ͞Δ͜ͱʹؔ܎͢ΔҎԼͷ࡞ۀ wख࡞ۀͰ͋Δ͜ͱ w܁Γฦ͞ΕΔ͜ͱ wࣗಈԽͰ͖Δ͜ͱ wઓज़తͰ͋Δ͜ͱ w௕ظతͳՁ஋Λ࣋ͨͳ͍͜ͱ wαʔϏεͷ੒௕ʹରͯ͠0
O Ͱ͋Δ͜ͱ ˞ষʮτΠϧʯͷ๾໓ΑΓҾ༻

ষʮτΠϧʯͷ๾໓ z௨ৗӡ༻தͷγεςϜʹਓख͕ඞཁͳΒɺ  ͦΕ͸όάͩɻ ௨ৗͷఆٛ͸ɺγεςϜͷ੒௕ͱڞʹมԽ͢Δɻ $BSMB(FJTTFSɺ(PPHMF43&z

ӡ༻ࣗಈԽ ࿑ಇ࣌ؒ୹ॖɾ඼࣭޲্ɾίετ࡟ݮ

ࣗಈԽͷ೉͠͞ w࣮ߦεέδϡʔϦϯά w࣮ߦঢ়ଶɾ࣮ߦ݁ՌͷՄࢹԽ wࣦഊ࣌ͷ࠶ࢼߦ ϦτϥΠ wλΠϜΞ΢τॲཧ wґଘؔ܎ͷ͋ΔλεΫ
wҟৗܥϑϩʔ wϩΪϯά wฒྻԽ w੒ޭɾࣦഊͷ݁Ռ௨஌ wʜ ର৅ͷपลͰߟྀ͢΂͖ࣄ߲͕ଟ͍

·ͱΊ

PQTXJUDI Φϓε΢Οον

ࣗಈԽͰ͖ΔλεΫ܈ w &$όοΫΞοϓͷ࡞੒ w &$Πϯελϯεͷىಈɾఀࢭ w &$ΠϯελϯελΠϓͷมߋ w
&$όοΫΞοϓͷϦʔδϣϯؒίϐʔ w &#4εφοϓγϣοτͷ࡞੒ w &#4εφοϓγϣοτͷϦʔδϣϯؒίϐʔ w 3%4εφοϓγϣοτͷ࡞੒ w 3%4Πϯελϯεͷ࡟আ w 3%4εφοϓγϣοτ͔Βͷ෮ݩ w 3%4Πϯελϯεͷىಈɾఀࢭ w ϦιʔεͷࢭΊ๨ΕνΣοΫ w ηΩϡϦςΟνΣοΫ w ʜਵ࣌௥Ճத

δϣϒػೳ ༻్ʹԠͯ͡ॊೈʹࢦఆՄೳͳ࣮ߦεέδϡʔϧ &$όοΫΞοϓͱ &#4εφοϓγϣοτɺ 3%4ͷ%#εφοϓγϣοτ࡞੒ ͳͲɺࣗ༝ͳλεΫͷ૊Έ߹Θͤ δϣϒͷ࣮ߦཤྺɺ ॲཧ࣌ؒɺϩάͷ֬ೝ

"84ӡ༻ࣗಈԽαʔϏεͷཪଆ αʔϏεͷϕʔεͱͳΔ  "NB[PO&$4ͱ"QBDIF"JSqPX

"NB[PO&$4 &MBTUJD$POUBJOFS4FSWJDF %PDLFSίϯςφΛ؆୯ʹӡ༻ɾ؅ཧͰ͖ΔίϯςφΦʔέετϨʔγϣϯαʔϏε IUUQTBXTBNB[PODPNKQFDT

"QBDIF"JSqPX w044ͷ1ZUIPO੡ϫʔΫϑϩʔΤϯδϯ "QBDIF-JDFOTF w1ZUIPOίʔυͰදݱͨ͠ϫʔΫϑϩʔ %"( ΛεέδϡʔϦϯά ˍϞχλϦϯά͢ΔϓϥοτϑΥʔϜ 
8PSLqPXBTB$PEF wෳ਺ͷλεΫ 0QFSBUPS ͷ࣮ߦॱংΛఆٛ͢ΔϫʔΫϑϩʔͷ࡞ ੒ɺεέδϡʔϦϯάɺϞχλϦϯάɺϦτϥΠɺ௨஌ͳͲ͕Մೳ "JSqPX͕ࣗಈԽͷෳࡶ͞Λαϙʔτ͢ΔͨΊɺ  ɹɹࢲͨͪ͸ۀ຿ϩδοΫͷ։ൃʹ஫ྗͰ͖Δ

"QBDIF"JSqPX4BB4 $MPVE$PNQPTFS ϑϧϚωʔδυ"JSqPX "TUSPOPNFS$MPVE&OUFSQSJTF "JSqPXBTB4FSWJDF

"QBDIF"JSqPXͷߏ੒ཁૉ 5BTL" 5BTL# 5BTL$ 5BTL% 5BTL& 5BTL' %"(
w0QFSBUPS  λεΫͷςϯϓϨʔτ w5BTL  0QFSBUPS͕Πϯελϯ εԽ͞Εͨ΋ͷ w%"(  ϫʔΫϑϩʔɺλεΫ ͷґଘؔ܎  1ZUIPOϑΝΠϧͰఆٛ ˞%"(%JSFDUFE"DZDMJD(SBQI

"QBDIF"JSqPXͷߏ੒ཁૉ

"QBDIF"JSqPXͷཁૉ Web UI / ؅ཧίϯιʔϧΛఏڙ͢Δ

"QBDIF"JSqPXͷཁૉ δϣϒ(DAG)ͷεέδϡʔϧΛ࣮ߦ͢Δ

"QBDIF"JSqPXͷཁૉ DAGͰఆٛ͞ΕͨλεΫΛ࣮ߦ͢Δ

αʔϏεͷཪଆΛ঺հ͢Δʹ͋ͨΓେࣄͳ͜ͱ ͜Ε͕ϕετͱ͍͏Θ͚Ͱ͸͋Γ·ͤΜɻ ઃܭ౰࣌ͷʮཁ݅ʯʮঢ়گʯʮ੍໿ʯͳͲʹ ΑΓબ୒͞Εͨ΋ͷͰ͋Γվળͷ༨஍͕͋Δ ΋ͷͰ͢ɻ

શମΞʔΩςΫνϟ

ίΞ෦෼ʹߜͬͨΞʔΩςΫνϟ

σϓϩΠͷ࢓૊Έ

%"(ϑΝΠϧετϨʔδͱͯ͠ͷ4 "JSqPX༻4όέοτ wEBHT w૊৫" wEBHϑΝΠϧ" wEBHϑΝΠϧ# w૊৫# wEBHϑΝΠϧ$
wEBHϑΝΠϧ% wEBHϑΝΠϧ& wʜ ӬଓԽͱϊʔυؒͰͷڞ༗ͷͨΊ4΁

%"(ϑΝΠϧͷ࣮૷ # -*- coding: utf-8 -*- import time
from pprint import pprint import airflow from airflow.models import DAG from airflow.operators.python_operator import PythonOperator, PythonVirtualenvOperator args = { 'owner': 'Airflow', 'start_date': airflow.utils.dates.days_ago(2), } dag = DAG( dag_id='example_python_operator', default_args=args, schedule_interval=None, ) def print_context(ds, **kwargs): pprint(kwargs) print(ds) return 'Whatever you return gets printed in the logs' run_this = PythonOperator( task_id='print_the_context', python_callable=print_context, dag=dag, ) def my_sleeping_function(random_base): time.sleep(random_base) for i in range(5): task = PythonOperator( task_id='sleep_for_' + str(i), python_callable=my_sleeping_function, op_kwargs={'random_base': float(i) / 10}, dag=dag, ) run_this >> task … https://github.com/apache/airﬂow/blob/master/airﬂow/example_dags/example_python_operator.py

֓೦σʔλϞσϧ

"84ӡ༻δϣϒͷ࣮ߦ"84ΞΧ΢ϯτ࿈ܞ "844FDVSJUZ5PLFO4FSWJDF 454 "TTVNF3PMF"1* Ұ࣌తͳηΩϡϦςΟೝূ৘ใΛऔಘɻ  ͜ΕΒΛ༻͍ͯɺڐՄ͞ΕͨผͷΞΧ΢ϯτ ͷϦιʔεʹΞΫηεՄೳɻ
w ΞΫηεΩʔ*% w γʔΫϨοτΞΫηεΩʔ w ηογϣϯτʔΫϯ

"84ΞΧ΢ϯτ࿈ܞʹΑΔΞΫηε ଞͷ"84ΞΧ΢ϯτ͔Β͸ "TTVNF3PMFͰ͖·ͤΜɻ ֎෦*%͕ඞཁͳͷͰɺͦΕΛ ࣋ͨͳ͍ଞͷ"84ΞΧ΢ϯτ ͸"TTVNF3PMF͸Ͱ͖·ͤΜɻ δϣϒͷ࣮ߦʹඞཁͳϦιʔε΁ͷΞΫηεݖݶͷΈ ෇༩͠·͢ɻ4΍σʔλϕʔεʹ֨ೲ͞ΕͨσʔλΛ
Ӿཡ͢Δ͜ͱ͸Ͱ͖·ͤΜɻ

ϝʔϧ഑৴ "NB[PO4&4 "NB[PO1JOQPJOU ݸผૹ৴ w Ϣʔβʔొ࿥ w Ϣʔβʔট଴
w δϣϒ࣮ߦ௨஌ w ʜ ෳ਺ૹ৴ w ͓஌Βͤ഑৴ w ϝϯςφϯε௨஌ w ʜ

"NB[PO1JOQPJOU ෳ਺ͷϝοηʔδϯάνϟωϧΛ௨ͯ͡ސ٬ͱίϛϡχέʔγϣϯ͢ΔαʔϏε IUUQTBXTBNB[PODPNKQQJOQPJOU

ϝʔϧ഑৴ͱό΢ϯεϝʔϧରԠ

γεςϜϞχλϦϯά

γεςϜϞχλϦϯά.BDLFSFMμογϡϘʔυ

γεςϜϞχλϦϯά.BDLFSFM4MBDL௨஌

γεςϜϞχλϦϯά಺੡ӡ༻πʔϧ

,1*ϞχλϦϯά "NB[PO2VJDL4JHIUͰՄࢹԽˠ

,1*ϞχλϦϯά ि࣍Ͱ4MBDL΁ϨϙʔςΟϯάˠ

Πϯϑϥ؀ڥ܈ ։ൃ؀ڥ ෛՙςετ  ؀ڥ εςʔδϯά  ؀ڥ ຊ൪؀ڥ ϩʔΧϧ 
։ൃ؀ڥ  EPDLFSDPNQPTF "84ΞΧ΢ϯτ9 "84ΞΧ΢ϯτ: "84ΞΧ΢ϯτ;

ϩʔΧϧ։ൃ؀ڥͷ࢓૊Έ $ docker-compose up -d

%PDLFS$PNQPTF WFSTJPO TFSWJDFT SFEJT JNBHFSFEJT QPTUHSFT JNBHFQPTUHSFT
XFCTFSWFS CVJME DPOUFYU\18%^ EPDLFSpMF\18%^%PDLFSpMF JNBHFPQTXJUDI DPNNBOEXFCTFSWFS TDIFEVMFS JNBHFPQTXJUDI DPNNBOETDIFEVMFS XPSLFS JNBHFPQTXJUDI DPNNBOEXPSLFS WPMVNFT ECEBUB docker-compose.yml ෳ਺ͷίϯςφΛ࢖͏ DockerΞϓϦέʔγϣϯΛఆٛ͠ɺ ૢ࡞͢Δπʔϧ

·ͱΊ

͓࿩ͨ͜͠ͱ wഎܠͱͳΔ"84ӡ༻ͱ՝୊ wτΠϧͷ๾໓ɺࣗಈԽͷͨΊͷߟྀ఺ͷଟ͞ w"84ӡ༻ࣗಈԽαʔϏεͷཪଆΛ࣌ؒ಺ͰՄೳͳݶΓ঺հ wࣗಈԽͷෳࡶ͞ʹରԠ͢ΔͨΊͷ"NB[PO&$4ͱ"QBDIF "JSqPXΛ૊Έ߹Θͤ wΞʔΩςΫνϟ͔ΒσϓϩΠɺϝʔϧ഑৴ɺϞχλϦϯάɺ ։ൃ·Ͱ

ηογϣϯͷ໨తئ๬ ঺հ͢Δଟ͘ͷ৘ใͷத͔ΒɺҰͭͰ΋ ։ൃɾӡ༻ͷώϯτʹͳΔ΋ͷΛ͝ఏڙ͢Δ

Amazon ECSを活用したAWS運用自動化サービスの裏側を包み隠さず解説 / inside opswitch

Amazon ECSを活用したAWS運用自動化サービスの裏側を包み隠さず解説 / inside opswitch

More Decks by shoito

Other Decks in Technology

Featured

Transcript