toward airflow without pain

Kenji Yamauchi
Machine Learning Production Pitch #2
https://machine-learning-pitch.connpass.com/event/125115/

Transcript

1. ͭΒ͘ͳ͍"JSGMPXʹΉ͚ͯ .BDIJOF
   -FBSOJOH
   1SPEVDUJPO
   1JUDI
    .BZ
   UI
    ,FOKJ
   :BNBVDIJ
   !
   -*/&
   $PSQ
2. ࣗݾ঺հ ,FOKJ
   :BNBVDIJ
   ࢁ಺ ݈ೋ • 4PGUXBSF
   &OHJOFFS
   !
   -*/&
   $MPWB 7"։ൃνʔϜ • Ի੠ೝࣝνʔϜ • ᑹܗจࣈͱ͔͕޷͖Ͱ͢
3. ࠓ೔࿩͍ͨ͜͠ͱ • ॴଐνʔϜͰͷ"JSGMPXΛ༻͍ͨ.-ϫʔΫϑϩʔ։ൃ ӡ༻ঢ়گ • ݱߦͷ"JSGMPX؀ڥͷ࣮૷ɾӡ༻ͷվળʹ޲͚ͯ ߦ͍ͬͯΔ಺༰
4. "HFOEB • ݱߦͷ"JSGMPX؀ڥ • ؀ڥվળʹ޲͚ͨऔΓ૊Έ • ΞʔΩςΫνϟվળ • %"(ʹର͢Δςετͷಋೖ
5. ݱߦͷ"JSGMPX؀ڥ
6. "JSGMPX • 044ͷXPSLGMPXΤϯδϯ • 1ZUIPOͰ%"( ༗޲ඇ८ճάϥϑ Ͱґଘ͢Δ5BTL܈ ͷؔ܎Λఆٛͯ͠XPSLGMPXΛ࣮૷ • ඪ४(6*͔ΒͰ͖ΔɾՄࢹԽͰ͖Δ߲໨͕ଟ͍ • 1MVHJOͰ(6*΋ؚΊ֦ͯு͕Ͱ͖ΔͷͰɺ ଍Γͳ͍ػೳ͕͋ͬͯ΋࠷ѱࣗ࡞Ͱ͖Δ
7. ಋೖ·ͰͷܦҢ • ৽͘͠.-Ϟσϧͷσʔλ੔ܗʙQVCMJTI·ͰΛ แׅతʹ࣮૷͢ΔҊ͕݅ൃੜ • ཁ݅తʹґଘੑͷ͋Δ5BTLΛ૊Έ্࣮͛ͯ૷͢Δඞཁ ͕͋ΓɺXPSLGMPXΤϯδϯͷಋೖΛܾఆ 8PSLGMPXྫ • िͰSBXσʔλΛऔಘͯ͠੔ܗޙֶशʙධՁ • Ϟσϧͷ࣮૷͕มΘͬͯ΋ֶशʙධՁ • ϞσϧΛར༻͍ͯ͠Δπʔϧͷมߋ͕͋ͬͯ΋&&ͰධՁ
8. ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend)
   
   Workers Web Server Celery Scheduler Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment
9. ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend) Workers Web Server Celery Scheduler Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment M G Q AD Q PS Q
10. ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend)  Workers Web Server Celery Scheduler Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment 
    
        
11. ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend)  Workers Web Server Celery Scheduler Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment 
       
12. %"(ͷ࣮૷ͷجຊํ਑ • ֶश΍ධՁ͸࣮૷Λ%"(ʹॻ͔ͳ͍Α͏ʹ෼཭͠ɺ %"(Ͱ͸ͦΕΛ࣮ߦ͢Δ͚ͩʹ͢Δ • ۩ମతͳϩδοΫΛ֎ग़͢͠Δ͜ͱͰɺ 5BTL΍෼ذͷ਺͕গͳ͘ͳΓɺ؅ཧָ͕ʹͳΔ • ֶश΍ධՁࣗମΛผͰςετͰ͖Δ ࣮࣭ਓ ඇઐଐ Ͱͷ؅ཧͷͨΊͳΔ͚ͩ୯७ʹ
13. ϏϧυʙධՁ·Ͱͷϑϩʔྫ ੔ܗ Models ֶश ධՁ ఆظతʹ SBXσʔλΛGFUDI ίʔυมԽʹUSJHHFS ࠷৽σʔλͰ ֶशΛUSJHHFS ֶशऴྃ࣌ʹUSJHHFS • ֶशɾධՁʹ͍ͭͯ͸ɺQPPMΛ͔࣮ͭͬͯߦϚγϯΛ %"(୯ҐͰࢦఆ ୆Ͱฒྻ࣮ߦ ݁Ռ௨஌ ϞσϧͷετΞ ߴεϖοΫϚγϯͰ࣮ߦ ʜ
    %"(
14. ݱߦͷ՝୊ • Ϧιʔε؅ཧͷίετ͕େ͖͍ ୆Ҏ্ͷ7.Πϯελϯε؅ཧ͕ඞཁͰɺ ֤Πϯελϯεͷঢ়ଶ͕Ұகͯ͠Δ͔΋ෆ҆ • %"(ͷ࣮ߦϚγϯࢦఆΛ΋ͬͱࡉ͔͘ߦ͍͍ͨ QPPM΍RVFVFͰ͋Δఔ౓࣮ݱՄೳ͕ͩɺϚγϯͷε ϖοΫʹ߹Θͤͨಈతͳࢦఆ͸͠ʹ͍͘ • %"(ʹ͍ͭͯςετίʔυ͕ͳ͘ɺ खܰͳϩʔΧϧςετ؀ڥ΋༻ҙ͠ʹ͍͘
15. ΞʔΩςΫνϟͷվળ
16. վળ໨ඪ • ϚγϯϦιʔε؅ཧίετͷ࡟ݮ Πϯϑϥߏ੒ͷߋ৽΍ҡ࣋ΛͳΔ͚ͩϑϨʔϜϫʔΫʹ౤͛ ͍ͨ • ؀ڥͷ࠶ݱੑͷ޲্ ίϯςφʹΑΔߏ੒؀ڥ౷Ұ • ϩʔΧϧ։ൃ؀ڥͷߏங؆қԽ
17. վળ໨ඪ • ϚγϯϦιʔε؅ཧίετͷ࡟ݮ Πϯϑϥߏ੒ͷߋ৽΍ҡ࣋ΛͳΔ͚ͩϑϨʔϜϫʔΫʹ౤͛ ͍ͨ • ؀ڥͷ࠶ݱੑͷ޲্ ίϯςφʹΑΔߏ੒؀ڥ౷Ұ • ϩʔΧϧ։ൃ؀ڥͷߏங؆қԽ è &
18. ,VCFSOFUFT • ίϯςφΦʔέετϨʔγϣϯϑϨʔϜϫʔΫ • Πϯϑϥߏ੒Λ:".-ͰએݴతʹఆٛՄೳ • "GGJOJUZͱ͍͏ػೳͰɺXPSLFSϚγϯͰͷ 1PE
    ࠓճ͸࣮࣭ίϯςφ
    ࣮ߦΛࡉ͔͘ઃఆՄೳ • "JSGMPX΋࣮ߦ؀ڥͱͯ͠ެࣜରԠ • 
    %"(
    
    
    1PE
    Ͱ࣮ߦ
19. %SPOF • 044ͷ$*$%πʔϧ • ίϯςφϕʔεͰͷϏϧυɾσϓϩΠ͕લఏͰɺ ಋೖ΋ൺֱత؆୯ • ฐࣾͰͷඪ४πʔϧͰ΋͋ΔͨΊ࣮੷͕͋Δ ࢀߟ
    IUUQTFOHJOFFSJOHMJOFDPSQDPNKBCMPHHPPTTDJUPPM ESPOFSFQMBDFTKFOLJOT • ίϯςφϏϧυͱ%"(ͷςετ ޙड़ ࣮ࢪʹར༻
20. ໨ඪ PostgreSQL (Airflow Backend)
    
    DAG Execution Pods Launches when DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Build Test Publish Deploy (Reload web server & scheduler images) Triggered by developer’s push Scheduler Airflow
21. ໨ඪ PostgreSQL (Airflow Backend)
    
    DAG Execution Pods Launches when DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Deploy (Reload web server & scheduler images) Scheduler Airflow Build Test Publish Triggered by developer’s push ࣮ߦ୯Ґ͸7.Πϯελϯε͔Β1PEʹ %"(΋ؚΊͯ"JSGMPX͸શ෦ͭʹ
22. ໨ඪ PostgreSQL (Airflow Backend)
    
    DAG Execution Pods Launches when DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Build Test Publish Deploy (Reload web server & scheduler images) Triggered by developer’s push Scheduler Airflow %"(΍%PDLFSGJMFʹมߋ͕͋Δ౓ʹ %SPOFͰϏϧυɾςετ࣮ߦ
23. ໨ඪ PostgreSQL (Airflow Backend)
    
    DAG Execution Pods Launches when DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Deploy (Reload web server & scheduler images) Scheduler Airflow Build Test Publish Triggered by developer’s push Ϗϧυ͕੒ޭͨ͠Β ϨδετϦʹίϯςφΛొ࿥"JSGMPX
    SFMPBE
24. ΞʔΩςΫνϟมߋʹΑΔվળ • ϚγϯϦιʔε؅ཧίετ࡟ݮ • ݸʑͷ7.Πϯελϯεͷ໘౗Λݟͳͯ͘Α͘ͳͬͨ • 8FCTFSWFS
    4DIFEVMFS
    8PSLFS͸શͯಉ͡࢓༷ͷίϯς φͳͷͰɺ%PDLFSGJMFͭΛ؅ཧ͢Ε͹ྑ͘ͳͬͨ • ͍ͭͰʹ$FMFSZΛ΍ΊͨͷͰ3FEJT؅ཧ΋ͳ͠ʹ • ίϯςφϕʔεͷ؅ཧʹͳΓɺ؀ڥ࠶ݱੑ΋޲্ • .JOJLVCFΛར༻ͨ͠ϩʔΧϧͰͷ؀ڥ࠶ݱ΋ָʹ
25. %"(ʹର͢Δςετͷಋೖ
26. %"(ͷςετͰ΍Γ͍ͨ͜ͱ • ཧ૝తʹ͸FOEUPFOEͰ%"(ͷڍಈΛอূ͍ͨ͠ • ࠷௿Ͱ΋5BTL୯ҐͰͷςετ͸͍ͨ͠ ྫ ֶश%"(ʹ͍ͭͯ • &&ςετͰ͸ֶश։࢝τϦΨ σʔλ੔ܗऴྃɾϞσϧ ίʔυมߋ ͝ͱʹਖ਼͘͠ىಈ͢Δ͔ɺֶशͷ࠷ޙʹਖ਼͍͠ ϑΥʔϚοτͷϑΝΠϧ໊ͰϞσϧΛग़ྗ͢Δ͔ FUD • 5BTL୯ҐͰ͸ɺਖ਼͍͠όʔδϣϯͷίʔυΛDMPOFͯ͜͠ Α͏ͱ͍ͯ͠Δ͔ FUD
27. ΍Ζ͏ͱ͍ͯ͠Δ %"(ςετͷ֓ཁ • ࣮૷ͨ͠5BTL΍%"(Λ࣮ߦ͢ΔΑ͏ͳVOJUUFTUΛॻ ͍ͯ(JU)VCʹQVTI͕͋Δ౓ʹ࣮ߦ͢Δ • %SPOF্Ͱ.JOJLVCFͳͲΛ࢖ͬͯμϛʔͷLT؀ڥΛ ࢖ͬͨςετ͸؆୯ʹ͸Ͱ͖ͳͦ͞͏ Ͱ͖ͦ͏Ͱͨ͠Βڭ͍͑ͯͩ͘͞
28. ςετͷॻ͖ํ 5BTL • μϛʔͷ%"(ΦϒδΣΫτʹର৅ͷ5BTLΛొ࿥͠ɺ ͦΕΛ࣮ߦ͢Δ͜ͱͰςετՄೳ • 5BTL͕ར༻͍ͯ͠Δ֎෦"1*Λ.PDL͓ͯ͘͠ / / . /- . . . / ςετର৅ͷλεΫ μϛʔ%"(ΦϒδΣΫτ μϛʔ%"(ΦϒδΣΫτ ʹΑΔ5BTL࣮ߦͷྫ
29. ςετͷॻ͖ํ && • "JSGMPXͷ%"(ͷ࣮ߦ࣌ͷ࣮ମͱͯ͠ %BH3VOΦϒδΣΫτΛੜ੒͢Δ • &&Ͱςετ͚ͨ͠Ε͹ɺ͜ͷ%BH3VOΦϒδΣΫτ ΛࣗલͰ࡞੒͢Ε͹Α͍ • "JSGMPXެࣜͷ%BH3VO5FTUʹଘࡏ͢Δ DSFBUF@EBH@SVOϔϧύʔؔ਺ͷ࣮૷͕ࢀߟʹͳΔ - 1/ - .1 .1 . /1 . 0