Slide 1

Slide 1 text

ͭΒ͘ͳ͍"JSGMPXʹΉ͚ͯ .BDIJOF-FBSOJOH1SPEVDUJPO1JUDI .BZUI  ,FOKJ:BNBVDIJ!-*/&$PSQ

Slide 2

Slide 2 text

ࣗݾ঺հ ,FOKJ:BNBVDIJ ࢁ಺ ݈ೋ • 4PGUXBSF&OHJOFFS!-*/&$MPWB 7"։ൃνʔϜ • Ի੠ೝࣝνʔϜ • ᑹܗจࣈͱ͔͕޷͖Ͱ͢

Slide 3

Slide 3 text

ࠓ೔࿩͍ͨ͜͠ͱ • ॴଐνʔϜͰͷ"JSGMPXΛ༻͍ͨ.-ϫʔΫϑϩʔ։ൃ ӡ༻ঢ়گ • ݱߦͷ"JSGMPX؀ڥͷ࣮૷ɾӡ༻ͷվળʹ޲͚ͯ ߦ͍ͬͯΔ಺༰

Slide 4

Slide 4 text

"HFOEB • ݱߦͷ"JSGMPX؀ڥ • ؀ڥվળʹ޲͚ͨऔΓ૊Έ • ΞʔΩςΫνϟվળ • %"(ʹର͢Δςετͷಋೖ

Slide 5

Slide 5 text

ݱߦͷ"JSGMPX؀ڥ

Slide 6

Slide 6 text

"JSGMPX • 044ͷXPSLGMPXΤϯδϯ • 1ZUIPOͰ%"( ༗޲ඇ८ճάϥϑ Ͱґଘ͢Δ5BTL܈ ͷؔ܎Λఆٛͯ͠XPSLGMPXΛ࣮૷ • ඪ४(6*͔ΒͰ͖ΔɾՄࢹԽͰ͖Δ߲໨͕ଟ͍ • 1MVHJOͰ(6*΋ؚΊ֦ͯு͕Ͱ͖ΔͷͰɺ ଍Γͳ͍ػೳ͕͋ͬͯ΋࠷ѱࣗ࡞Ͱ͖Δ

Slide 7

Slide 7 text

ಋೖ·ͰͷܦҢ • ৽͘͠.-Ϟσϧͷσʔλ੔ܗʙQVCMJTI·ͰΛ แׅతʹ࣮૷͢ΔҊ͕݅ൃੜ • ཁ݅తʹґଘੑͷ͋Δ5BTLΛ૊Έ্࣮͛ͯ૷͢Δඞཁ ͕͋ΓɺXPSLGMPXΤϯδϯͷಋೖΛܾఆ 8PSLGMPXྫ • िͰSBXσʔλΛऔಘͯ͠੔ܗޙֶशʙධՁ • Ϟσϧͷ࣮૷͕มΘͬͯ΋ֶशʙධՁ • ϞσϧΛར༻͍ͯ͠Δπʔϧͷมߋ͕͋ͬͯ΋&&ͰධՁ

Slide 8

Slide 8 text

ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend)  Workers Web Server Celery Scheduler Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment

Slide 9

Slide 9 text

ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend) Workers Web Server Celery Scheduler Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment M G Q AD Q PS Q

Slide 10

Slide 10 text

ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend)  Workers Web Server Celery Scheduler Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment       

Slide 11

Slide 11 text

ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend)  Workers Web Server Celery Scheduler Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment    

Slide 12

Slide 12 text

%"(ͷ࣮૷ͷجຊํ਑ • ֶश΍ධՁ͸࣮૷Λ%"(ʹॻ͔ͳ͍Α͏ʹ෼཭͠ɺ %"(Ͱ͸ͦΕΛ࣮ߦ͢Δ͚ͩʹ͢Δ • ۩ମతͳϩδοΫΛ֎ग़͢͠Δ͜ͱͰɺ 5BTL΍෼ذͷ਺͕গͳ͘ͳΓɺ؅ཧָ͕ʹͳΔ • ֶश΍ධՁࣗମΛผͰςετͰ͖Δ ࣮࣭ਓ ඇઐଐ Ͱͷ؅ཧͷͨΊͳΔ͚ͩ୯७ʹ

Slide 13

Slide 13 text

ϏϧυʙධՁ·Ͱͷϑϩʔྫ ੔ܗ Models ֶश ධՁ ఆظతʹ SBXσʔλΛGFUDI ίʔυมԽʹUSJHHFS ࠷৽σʔλͰ ֶशΛUSJHHFS ֶशऴྃ࣌ʹUSJHHFS • ֶशɾධՁʹ͍ͭͯ͸ɺQPPMΛ͔࣮ͭͬͯߦϚγϯΛ %"(୯ҐͰࢦఆ ୆Ͱฒྻ࣮ߦ ݁Ռ௨஌ ϞσϧͷετΞ ߴεϖοΫϚγϯͰ࣮ߦ ʜ%"(

Slide 14

Slide 14 text

ݱߦͷ՝୊ • Ϧιʔε؅ཧͷίετ͕େ͖͍ ୆Ҏ্ͷ7.Πϯελϯε؅ཧ͕ඞཁͰɺ ֤Πϯελϯεͷঢ়ଶ͕Ұகͯ͠Δ͔΋ෆ҆ • %"(ͷ࣮ߦϚγϯࢦఆΛ΋ͬͱࡉ͔͘ߦ͍͍ͨ QPPM΍RVFVFͰ͋Δఔ౓࣮ݱՄೳ͕ͩɺϚγϯͷε ϖοΫʹ߹Θͤͨಈతͳࢦఆ͸͠ʹ͍͘ • %"(ʹ͍ͭͯςετίʔυ͕ͳ͘ɺ खܰͳϩʔΧϧςετ؀ڥ΋༻ҙ͠ʹ͍͘

Slide 15

Slide 15 text

ΞʔΩςΫνϟͷվળ

Slide 16

Slide 16 text

վળ໨ඪ • ϚγϯϦιʔε؅ཧίετͷ࡟ݮ Πϯϑϥߏ੒ͷߋ৽΍ҡ࣋ΛͳΔ͚ͩϑϨʔϜϫʔΫʹ౤͛ ͍ͨ • ؀ڥͷ࠶ݱੑͷ޲্ ίϯςφʹΑΔߏ੒؀ڥ౷Ұ • ϩʔΧϧ։ൃ؀ڥͷߏங؆қԽ

Slide 17

Slide 17 text

վળ໨ඪ • ϚγϯϦιʔε؅ཧίετͷ࡟ݮ Πϯϑϥߏ੒ͷߋ৽΍ҡ࣋ΛͳΔ͚ͩϑϨʔϜϫʔΫʹ౤͛ ͍ͨ • ؀ڥͷ࠶ݱੑͷ޲্ ίϯςφʹΑΔߏ੒؀ڥ౷Ұ • ϩʔΧϧ։ൃ؀ڥͷߏங؆қԽ è &

Slide 18

Slide 18 text

,VCFSOFUFT • ίϯςφΦʔέετϨʔγϣϯϑϨʔϜϫʔΫ • Πϯϑϥߏ੒Λ:".-ͰએݴతʹఆٛՄೳ • "GGJOJUZͱ͍͏ػೳͰɺXPSLFSϚγϯͰͷ 1PE ࠓճ͸࣮࣭ίϯςφ ࣮ߦΛࡉ͔͘ઃఆՄೳ • "JSGMPX΋࣮ߦ؀ڥͱͯ͠ެࣜରԠ • %"(1PEͰ࣮ߦ

Slide 19

Slide 19 text

%SPOF • 044ͷ$*$%πʔϧ • ίϯςφϕʔεͰͷϏϧυɾσϓϩΠ͕લఏͰɺ ಋೖ΋ൺֱత؆୯ • ฐࣾͰͷඪ४πʔϧͰ΋͋ΔͨΊ࣮੷͕͋Δ ࢀߟIUUQTFOHJOFFSJOHMJOFDPSQDPNKBCMPHHPPTTDJUPPM ESPOFSFQMBDFTKFOLJOT • ίϯςφϏϧυͱ%"(ͷςετ ޙड़ ࣮ࢪʹར༻

Slide 20

Slide 20 text

໨ඪ PostgreSQL (Airflow Backend)  DAG Execution Pods Launches when DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Build Test Publish Deploy (Reload web server & scheduler images) Triggered by developer’s push Scheduler Airflow

Slide 21

Slide 21 text

໨ඪ PostgreSQL (Airflow Backend)  DAG Execution Pods Launches when DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Deploy (Reload web server & scheduler images) Scheduler Airflow Build Test Publish Triggered by developer’s push ࣮ߦ୯Ґ͸7.Πϯελϯε͔Β1PEʹ %"(΋ؚΊͯ"JSGMPX͸શ෦ͭʹ

Slide 22

Slide 22 text

໨ඪ PostgreSQL (Airflow Backend)  DAG Execution Pods Launches when DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Build Test Publish Deploy (Reload web server & scheduler images) Triggered by developer’s push Scheduler Airflow %"(΍%PDLFSGJMFʹมߋ͕͋Δ౓ʹ %SPOFͰϏϧυɾςετ࣮ߦ

Slide 23

Slide 23 text

໨ඪ PostgreSQL (Airflow Backend)  DAG Execution Pods Launches when DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Deploy (Reload web server & scheduler images) Scheduler Airflow Build Test Publish Triggered by developer’s push Ϗϧυ͕੒ޭͨ͠Β ϨδετϦʹίϯςφΛొ࿥"JSGMPXSFMPBE

Slide 24

Slide 24 text

ΞʔΩςΫνϟมߋʹΑΔվળ • ϚγϯϦιʔε؅ཧίετ࡟ݮ • ݸʑͷ7.Πϯελϯεͷ໘౗Λݟͳͯ͘Α͘ͳͬͨ • 8FCTFSWFS 4DIFEVMFS 8PSLFS͸શͯಉ͡࢓༷ͷίϯς φͳͷͰɺ%PDLFSGJMFͭΛ؅ཧ͢Ε͹ྑ͘ͳͬͨ • ͍ͭͰʹ$FMFSZΛ΍ΊͨͷͰ3FEJT؅ཧ΋ͳ͠ʹ • ίϯςφϕʔεͷ؅ཧʹͳΓɺ؀ڥ࠶ݱੑ΋޲্ • .JOJLVCFΛར༻ͨ͠ϩʔΧϧͰͷ؀ڥ࠶ݱ΋ָʹ

Slide 25

Slide 25 text

%"(ʹର͢Δςετͷಋೖ

Slide 26

Slide 26 text

%"(ͷςετͰ΍Γ͍ͨ͜ͱ • ཧ૝తʹ͸FOEUPFOEͰ%"(ͷڍಈΛอূ͍ͨ͠ • ࠷௿Ͱ΋5BTL୯ҐͰͷςετ͸͍ͨ͠ ྫ ֶश%"(ʹ͍ͭͯ • &&ςετͰ͸ֶश։࢝τϦΨ σʔλ੔ܗऴྃɾϞσϧ ίʔυมߋ ͝ͱʹਖ਼͘͠ىಈ͢Δ͔ɺֶशͷ࠷ޙʹਖ਼͍͠ ϑΥʔϚοτͷϑΝΠϧ໊ͰϞσϧΛग़ྗ͢Δ͔ FUD • 5BTL୯ҐͰ͸ɺਖ਼͍͠όʔδϣϯͷίʔυΛDMPOFͯ͜͠ Α͏ͱ͍ͯ͠Δ͔ FUD

Slide 27

Slide 27 text

΍Ζ͏ͱ͍ͯ͠Δ %"(ςετͷ֓ཁ • ࣮૷ͨ͠5BTL΍%"(Λ࣮ߦ͢ΔΑ͏ͳVOJUUFTUΛॻ ͍ͯ(JU)VCʹQVTI͕͋Δ౓ʹ࣮ߦ͢Δ • %SPOF্Ͱ.JOJLVCFͳͲΛ࢖ͬͯμϛʔͷLT؀ڥΛ ࢖ͬͨςετ͸؆୯ʹ͸Ͱ͖ͳͦ͞͏ Ͱ͖ͦ͏Ͱͨ͠Βڭ͍͑ͯͩ͘͞

Slide 28

Slide 28 text

ςετͷॻ͖ํ 5BTL • μϛʔͷ%"(ΦϒδΣΫτʹର৅ͷ5BTLΛొ࿥͠ɺ ͦΕΛ࣮ߦ͢Δ͜ͱͰςετՄೳ • 5BTL͕ར༻͍ͯ͠Δ֎෦"1*Λ.PDL͓ͯ͘͠ / / . /- . . . / ςετର৅ͷλεΫ μϛʔ%"(ΦϒδΣΫτ μϛʔ%"(ΦϒδΣΫτ ʹΑΔ5BTL࣮ߦͷྫ

Slide 29

Slide 29 text

ςετͷॻ͖ํ && • "JSGMPXͷ%"(ͷ࣮ߦ࣌ͷ࣮ମͱͯ͠ %BH3VOΦϒδΣΫτΛੜ੒͢Δ • &&Ͱςετ͚ͨ͠Ε͹ɺ͜ͷ%BH3VOΦϒδΣΫτ ΛࣗલͰ࡞੒͢Ε͹Α͍ • "JSGMPXެࣜͷ%BH3VO5FTUʹଘࡏ͢Δ DSFBUF@EBH@SVOϔϧύʔؔ਺ͷ࣮૷͕ࢀߟʹͳΔ - 1/ - .1 .1 . /1 . 0