Upgrade to Pro — share decks privately, control downloads, hide ads and more …

toward airflow without pain

1.7k

toward airflow without pain

Kenji Yamauchi
Machine Learning Production Pitch #2
https://machine-learning-pitch.connpass.com/event/125115/

53850955f15249a1a9dc49df6113e400?s=128

LINE Developers
PRO

May 30, 2019
Tweet

Transcript

  1. ͭΒ͘ͳ͍"JSGMPXʹΉ͚ͯ .BDIJOF-FBSOJOH1SPEVDUJPO1JUDI .BZUI  ,FOKJ:BNBVDIJ!-*/&$PSQ

  2. ࣗݾ঺հ ,FOKJ:BNBVDIJ ࢁ಺ ݈ೋ • 4PGUXBSF&OHJOFFS!-*/&$MPWB 7"։ൃνʔϜ • Ի੠ೝࣝνʔϜ •

    ᑹܗจࣈͱ͔͕޷͖Ͱ͢
  3. ࠓ೔࿩͍ͨ͜͠ͱ • ॴଐνʔϜͰͷ"JSGMPXΛ༻͍ͨ.-ϫʔΫϑϩʔ։ൃ ӡ༻ঢ়گ • ݱߦͷ"JSGMPX؀ڥͷ࣮૷ɾӡ༻ͷվળʹ޲͚ͯ ߦ͍ͬͯΔ಺༰

  4. "HFOEB • ݱߦͷ"JSGMPX؀ڥ • ؀ڥվળʹ޲͚ͨऔΓ૊Έ • ΞʔΩςΫνϟվળ • %"(ʹର͢Δςετͷಋೖ

  5. ݱߦͷ"JSGMPX؀ڥ

  6. "JSGMPX • 044ͷXPSLGMPXΤϯδϯ • 1ZUIPOͰ%"( ༗޲ඇ८ճάϥϑ Ͱґଘ͢Δ5BTL܈ ͷؔ܎Λఆٛͯ͠XPSLGMPXΛ࣮૷ • ඪ४(6*͔ΒͰ͖ΔɾՄࢹԽͰ͖Δ߲໨͕ଟ͍

    • 1MVHJOͰ(6*΋ؚΊ֦ͯு͕Ͱ͖ΔͷͰɺ ଍Γͳ͍ػೳ͕͋ͬͯ΋࠷ѱࣗ࡞Ͱ͖Δ
  7. ಋೖ·ͰͷܦҢ • ৽͘͠.-Ϟσϧͷσʔλ੔ܗʙQVCMJTI·ͰΛ แׅతʹ࣮૷͢ΔҊ͕݅ൃੜ • ཁ݅తʹґଘੑͷ͋Δ5BTLΛ૊Έ্࣮͛ͯ૷͢Δඞཁ ͕͋ΓɺXPSLGMPXΤϯδϯͷಋೖΛܾఆ 8PSLGMPXྫ • िͰSBXσʔλΛऔಘͯ͠੔ܗޙֶशʙධՁ

    • Ϟσϧͷ࣮૷͕มΘͬͯ΋ֶशʙධՁ • ϞσϧΛར༻͍ͯ͠Δπʔϧͷมߋ͕͋ͬͯ΋&&ͰධՁ
  8. ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend)  Workers Web Server Celery Scheduler

    Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment
  9. ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend) Workers Web Server Celery Scheduler Airflow

    Redis (Celery Broker) DAGs Playbooks Bucket Deployment M G Q AD Q PS Q
  10. ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend)  Workers Web Server Celery Scheduler

    Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment       
  11. ΞʔΩςΫνϟશମ PostgreSQL (Airflow Backend)  Workers Web Server Celery Scheduler

    Airflow Redis (Celery Broker) DAGs Playbooks Bucket Deployment    
  12. %"(ͷ࣮૷ͷجຊํ਑ • ֶश΍ධՁ͸࣮૷Λ%"(ʹॻ͔ͳ͍Α͏ʹ෼཭͠ɺ %"(Ͱ͸ͦΕΛ࣮ߦ͢Δ͚ͩʹ͢Δ • ۩ମతͳϩδοΫΛ֎ग़͢͠Δ͜ͱͰɺ 5BTL΍෼ذͷ਺͕গͳ͘ͳΓɺ؅ཧָ͕ʹͳΔ • ֶश΍ධՁࣗମΛผͰςετͰ͖Δ ࣮࣭ਓ

    ඇઐଐ Ͱͷ؅ཧͷͨΊͳΔ͚ͩ୯७ʹ
  13. ϏϧυʙධՁ·Ͱͷϑϩʔྫ ੔ܗ Models ֶश ධՁ ఆظతʹ SBXσʔλΛGFUDI ίʔυมԽʹUSJHHFS ࠷৽σʔλͰ ֶशΛUSJHHFS

    ֶशऴྃ࣌ʹUSJHHFS • ֶशɾධՁʹ͍ͭͯ͸ɺQPPMΛ͔࣮ͭͬͯߦϚγϯΛ %"(୯ҐͰࢦఆ ୆Ͱฒྻ࣮ߦ ݁Ռ௨஌ ϞσϧͷετΞ ߴεϖοΫϚγϯͰ࣮ߦ ʜ%"(
  14. ݱߦͷ՝୊ • Ϧιʔε؅ཧͷίετ͕େ͖͍ ୆Ҏ্ͷ7.Πϯελϯε؅ཧ͕ඞཁͰɺ ֤Πϯελϯεͷঢ়ଶ͕Ұகͯ͠Δ͔΋ෆ҆ • %"(ͷ࣮ߦϚγϯࢦఆΛ΋ͬͱࡉ͔͘ߦ͍͍ͨ QPPM΍RVFVFͰ͋Δఔ౓࣮ݱՄೳ͕ͩɺϚγϯͷε ϖοΫʹ߹Θͤͨಈతͳࢦఆ͸͠ʹ͍͘ •

    %"(ʹ͍ͭͯςετίʔυ͕ͳ͘ɺ खܰͳϩʔΧϧςετ؀ڥ΋༻ҙ͠ʹ͍͘
  15. ΞʔΩςΫνϟͷվળ

  16. վળ໨ඪ • ϚγϯϦιʔε؅ཧίετͷ࡟ݮ Πϯϑϥߏ੒ͷߋ৽΍ҡ࣋ΛͳΔ͚ͩϑϨʔϜϫʔΫʹ౤͛ ͍ͨ • ؀ڥͷ࠶ݱੑͷ޲্ ίϯςφʹΑΔߏ੒؀ڥ౷Ұ • ϩʔΧϧ։ൃ؀ڥͷߏங؆қԽ

  17. վળ໨ඪ • ϚγϯϦιʔε؅ཧίετͷ࡟ݮ Πϯϑϥߏ੒ͷߋ৽΍ҡ࣋ΛͳΔ͚ͩϑϨʔϜϫʔΫʹ౤͛ ͍ͨ • ؀ڥͷ࠶ݱੑͷ޲্ ίϯςφʹΑΔߏ੒؀ڥ౷Ұ • ϩʔΧϧ։ൃ؀ڥͷߏங؆қԽ

    è &
  18. ,VCFSOFUFT • ίϯςφΦʔέετϨʔγϣϯϑϨʔϜϫʔΫ • Πϯϑϥߏ੒Λ:".-ͰએݴతʹఆٛՄೳ • "GGJOJUZͱ͍͏ػೳͰɺXPSLFSϚγϯͰͷ 1PE ࠓճ͸࣮࣭ίϯςφ ࣮ߦΛࡉ͔͘ઃఆՄೳ

    • "JSGMPX΋࣮ߦ؀ڥͱͯ͠ެࣜରԠ • %"(1PEͰ࣮ߦ
  19. %SPOF • 044ͷ$*$%πʔϧ • ίϯςφϕʔεͰͷϏϧυɾσϓϩΠ͕લఏͰɺ ಋೖ΋ൺֱత؆୯ • ฐࣾͰͷඪ४πʔϧͰ΋͋ΔͨΊ࣮੷͕͋Δ ࢀߟIUUQTFOHJOFFSJOHMJOFDPSQDPNKBCMPHHPPTTDJUPPM ESPOFSFQMBDFTKFOLJOT

    • ίϯςφϏϧυͱ%"(ͷςετ ޙड़ ࣮ࢪʹར༻
  20. ໨ඪ PostgreSQL (Airflow Backend)  DAG Execution Pods Launches when

    DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Build Test Publish Deploy (Reload web server & scheduler images) Triggered by developer’s push Scheduler Airflow
  21. ໨ඪ PostgreSQL (Airflow Backend)  DAG Execution Pods Launches when

    DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Deploy (Reload web server & scheduler images) Scheduler Airflow Build Test Publish Triggered by developer’s push ࣮ߦ୯Ґ͸7.Πϯελϯε͔Β1PEʹ %"(΋ؚΊͯ"JSGMPX͸શ෦ͭʹ
  22. ໨ඪ PostgreSQL (Airflow Backend)  DAG Execution Pods Launches when

    DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Build Test Publish Deploy (Reload web server & scheduler images) Triggered by developer’s push Scheduler Airflow %"(΍%PDLFSGJMFʹมߋ͕͋Δ౓ʹ %SPOFͰϏϧυɾςετ࣮ߦ
  23. ໨ඪ PostgreSQL (Airflow Backend)  DAG Execution Pods Launches when

    DAG executes Web Server Kubernetes Cluster Store logs/training, evaluation data/models Docker Registry S3 like bucket Deploy (Reload web server & scheduler images) Scheduler Airflow Build Test Publish Triggered by developer’s push Ϗϧυ͕੒ޭͨ͠Β ϨδετϦʹίϯςφΛొ࿥"JSGMPXSFMPBE
  24. ΞʔΩςΫνϟมߋʹΑΔվળ • ϚγϯϦιʔε؅ཧίετ࡟ݮ • ݸʑͷ7.Πϯελϯεͷ໘౗Λݟͳͯ͘Α͘ͳͬͨ • 8FCTFSWFS 4DIFEVMFS 8PSLFS͸શͯಉ͡࢓༷ͷίϯς φͳͷͰɺ%PDLFSGJMFͭΛ؅ཧ͢Ε͹ྑ͘ͳͬͨ

    • ͍ͭͰʹ$FMFSZΛ΍ΊͨͷͰ3FEJT؅ཧ΋ͳ͠ʹ • ίϯςφϕʔεͷ؅ཧʹͳΓɺ؀ڥ࠶ݱੑ΋޲্ • .JOJLVCFΛར༻ͨ͠ϩʔΧϧͰͷ؀ڥ࠶ݱ΋ָʹ
  25. %"(ʹର͢Δςετͷಋೖ

  26. %"(ͷςετͰ΍Γ͍ͨ͜ͱ • ཧ૝తʹ͸FOEUPFOEͰ%"(ͷڍಈΛอূ͍ͨ͠ • ࠷௿Ͱ΋5BTL୯ҐͰͷςετ͸͍ͨ͠ ྫ ֶश%"(ʹ͍ͭͯ • &&ςετͰ͸ֶश։࢝τϦΨ σʔλ੔ܗऴྃɾϞσϧ

    ίʔυมߋ ͝ͱʹਖ਼͘͠ىಈ͢Δ͔ɺֶशͷ࠷ޙʹਖ਼͍͠ ϑΥʔϚοτͷϑΝΠϧ໊ͰϞσϧΛग़ྗ͢Δ͔ FUD • 5BTL୯ҐͰ͸ɺਖ਼͍͠όʔδϣϯͷίʔυΛDMPOFͯ͜͠ Α͏ͱ͍ͯ͠Δ͔ FUD
  27. ΍Ζ͏ͱ͍ͯ͠Δ %"(ςετͷ֓ཁ • ࣮૷ͨ͠5BTL΍%"(Λ࣮ߦ͢ΔΑ͏ͳVOJUUFTUΛॻ ͍ͯ(JU)VCʹQVTI͕͋Δ౓ʹ࣮ߦ͢Δ • %SPOF্Ͱ.JOJLVCFͳͲΛ࢖ͬͯμϛʔͷLT؀ڥΛ ࢖ͬͨςετ͸؆୯ʹ͸Ͱ͖ͳͦ͞͏ Ͱ͖ͦ͏Ͱͨ͠Βڭ͍͑ͯͩ͘͞

  28. ςετͷॻ͖ํ 5BTL • μϛʔͷ%"(ΦϒδΣΫτʹର৅ͷ5BTLΛొ࿥͠ɺ ͦΕΛ࣮ߦ͢Δ͜ͱͰςετՄೳ • 5BTL͕ར༻͍ͯ͠Δ֎෦"1*Λ.PDL͓ͯ͘͠ / / .

    /- . . . / ςετର৅ͷλεΫ μϛʔ%"(ΦϒδΣΫτ μϛʔ%"(ΦϒδΣΫτ ʹΑΔ5BTL࣮ߦͷྫ
  29. ςετͷॻ͖ํ && • "JSGMPXͷ%"(ͷ࣮ߦ࣌ͷ࣮ମͱͯ͠ %BH3VOΦϒδΣΫτΛੜ੒͢Δ • &&Ͱςετ͚ͨ͠Ε͹ɺ͜ͷ%BH3VOΦϒδΣΫτ ΛࣗલͰ࡞੒͢Ε͹Α͍ • "JSGMPXެࣜͷ%BH3VO5FTUʹଘࡏ͢Δ

    DSFBUF@EBH@SVOϔϧύʔؔ਺ͷ࣮૷͕ࢀߟʹͳΔ - 1/ - .1 .1 . /1 . 0