NLU Architecture and
 ML Model Management in Clova Machine Learning Production Pitch #1 March 7th, 2019 Yuki Matoba (LINE Corporation) 1

ࣗݾ঺հ • Yuki Matoba (త৔ ༐थ) • LINE Clova VA։ൃνʔϜɹ • Clova NLU system • Rekcurd • GitHub: @yuki-mt 2

ࠓ೔࿩͢͜ͱ มߋΛૉૣ͘ϦϦʔε͢ΔͨΊͷɺ Clovaʹ͓͚ΔNLU (ࣗવݴޠཧղ) γεςϜͷΞʔΩςΫνϟ΍ ػցֶशϞσϧͷֶशɺධՁɺӡ༻ํ๏Λ঺հ 3

Agenda • Clova, NLUपลͷΞʔΩςΫνϟͷ؆୯ͳ঺հ • NLUγεςϜͷΞʔΩςΫνϟ • ػցֶशϞσϧͷӡ༻ํ๏ 4

Clovaͷ঺հ 5

NLUपลͷΞʔΩςΫνϟ ϢʔβʔͷԻ੠͕ཧղ͞ΕɺԠ౴Λฦ͢·ͰͷྲྀΕ 6

Clova user’s speech data Ͷ͑ Clova, ࠓͷ৽॓ͷఱؾ͸? 7

Clova user’s speech data Speech Recognition user’s 
 speech data Ͷ͑ Clova, ࠓͷ৽॓ͷఱؾ͸? ݱࡏͷ৽॓ͷఱؾ͸੖Ε (The weather of the present Shinjuku is sunny) 8

Clova user’s speech data Speech Recognition recognized
 text list user’s 
 speech data Ͷ͑ Clova, ࠓͷ৽॓ͷఱؾ͸? ݱࡏͷ৽॓ͷఱؾ͸੖Ε (The weather of the present Shinjuku is sunny) ࠓͷ৽॓ͷ ఱؾ͸? 9

Clova NLU / DM recognized
 text list user’s speech data Speech Recognition recognized
 text list user’s 
 speech data Ͷ͑ Clova, ࠓͷ৽॓ͷఱؾ͸? ݱࡏͷ৽॓ͷఱؾ͸੖Ε (The weather of the present Shinjuku is sunny) ࠓͷ৽॓ͷ ఱؾ͸? 10

Clova Key Value Domain Weather Intention Inform Main Goal General Place Shinjuku Time Present NLU Result NLU / DM recognized
 text list user’s speech data Speech Recognition recognized
 text list user’s 
 speech data Ͷ͑ Clova, ࠓͷ৽॓ͷఱؾ͸? ݱࡏͷ৽॓ͷఱؾ͸੖Ε (The weather of the present Shinjuku is sunny) ࠓͷ৽॓ͷ ఱؾ͸? 11

Clova Weather web API generated text NLU
 Result NLU Result ࠓͷ৽॓ͷ ఱؾ͸? NLU / DM recognized
 text list user’s speech data Speech Recognition recognized
 text list user’s 
 speech data Ͷ͑ Clova, ࠓͷ৽॓ͷఱؾ͸? ݱࡏͷ৽॓ͷఱؾ͸੖Ε Key Value Domain Weather Intention Inform Main Goal General Place Shinjuku Time Present 12

Clova Speech Synthesis generated text Weather web API generated text NLU
 Result Key Value Domain Weather Intention Inform Main Goal General Place Shinjuku Time Present NLU Result NLU / DM recognized
 text list user’s speech data Speech Recognition recognized
 text list user’s 
 speech data Ͷ͑ Clova, ࠓͷ৽॓ͷఱؾ͸? ݱࡏͷ৽॓ͷఱؾ͸੖Ε ࠓͷ৽॓ͷ ఱؾ͸? 13

Clova Speech Synthesis synthesized 
 speech data synthesized speech data Weather web API generated text NLU
 Result NLU Result NLU / DM recognized
 text list user’s speech data Speech Recognition recognized
 text list user’s 
 speech data Ͷ͑ Clova, ࠓͷ৽॓ͷఱؾ͸? generated text ࠓͷ৽॓ͷ ఱؾ͸? Key Value Domain Weather Intention Inform Main Goal General Place Shinjuku Time Present 14

NLUγεςϜͷΞʔΩςΫνϟ γεςϜؒ, εςʔΫϗϧμʔؒΛૄ݁߹ʹ͢ΔͨΊͷߏ଄ 15

Clova Key Value Domain Weather Intention Inform Main Goal General Place Shinjuku Time Present NLU Result NLU / DM recognized
 text list ࠓͷ৽॓ͷ ఱؾ͸? 16 NLUγεςϜ͕͍ͯ͠Δ͜ͱ(͓͞Β͍) 1. ൃ࿩ͷจࣈྻΛड͚औΔ 2. ߏ଄Խ͞ΕͨσʔλΛฦ͢

NLUγεςϜͷ࢓༷มߋͷେม͞ Өڹൣғ΍ґଘؔ܎͕ෳࡶʹͳΓ΍͍͢ • ʮແݶύλʔϯͷൃ࿩ x ෳࡶͳcontextʯΛ ༷ʑͳDomainͰਖ਼͘͠෼୲͢Δඞཁ͕͋ΔͨΊ 17

NLUγεςϜͷ࢓༷มߋͷେม͞ • ྫ: ʮകӍ͕͍͔ͭڭ͑ͯʯͱ͍͏ൃ࿩Λweather domainͰ৽͘͠ ରԠ͍ͨ͠৔߹ɺ֬ೝ͢΂͖߲໨͸ • ৽͍͠ϞσϧͰʮകӍ͕͍͔ͭڭ͑ͯʯ͸৽͍͠࢓༷௨Γʹ൑ผͰ͖͍ͯΔ͔ • ʮകӍ͕͍͔ͭڭ͑ͯʯҎ֎ͷݴ͍ճ͠͸ͲΕ͚ͩΧόʔͰ͖͍ͯΔ͔ • Χόʔͨ͠ݴ͍ճ͠ʹผͷDomain, Intention, Main Goalͱ൑ఆ͢΂͖΋ͷ͸ೖ͍ͬͯ ͳ͍͔ • ͜ͷมߋ͸ผͷػೳʹӨڹΛ༩͑ͳ͍͔ • ଞͷػೳʹӨڹΛ༩͑Δ৔߹ɺͲͷcontextͷ࣌ʹͲͷൃ࿩ͰͲͷػೳ͕༏ઌ͞ΕΔͷ ͔ • … 18

NLUγεςϜͷ࢓༷มߋͷେม͞ • ྫ: ʮകӍ͕͍͔ͭڭ͑ͯʯͱ͍͏ൃ࿩Λweather domainͰ৽͘͠ ରԠ͍ͨ͠৔߹ɺ֬ೝ͢΂͖߲໨͸ • ৽͍͠ϞσϧͰʮകӍ͕͍͔ͭڭ͑ͯʯ͸৽͍͠࢓༷௨Γʹ൑ผͰ͖͍ͯΔ͔ • ʮകӍ͕͍͔ͭڭ͑ͯʯҎ֎ͷݴ͍ճ͠͸ͲΕ͚ͩΧόʔͰ͖͍ͯΔ͔ • Χόʔͨ͠ݴ͍ճ͠ʹผͷDomain, Intention, Main Goalͱ൑ఆ͢΂͖΋ͷ͸ೖ͍ͬͯ ͳ͍͔ • ͜ͷมߋ͸ผͷػೳʹӨڹΛ༩͑ͳ͍͔ • ଞͷػೳʹӨڹΛ༩͑Δ৔߹ɺͲͷcontextͷ࣌ʹͲͷൃ࿩ͰͲͷػೳ͕༏ઌ͞ΕΔͷ ͔ • … → ։ൃͷӨڹൣғ΍֬ೝࣄ߲͕ଟ͘ɺϦϦʔε·Ͱʹ͕͔͔࣌ؒΔ 19

NLUγεςϜͷ(Ұ෦ͷ)ΞʔΩςΫνϟ Domain Detector weather rule-based NLU Global Pattern NLU other domain rule-based NLU1 : ॲཧͷॱ൪ weather ML-based NLU other domain ML-based NLU1 other domain rule-based NLU2 other domain ML-based NLU2 weather domain-specific NLU 20

NLUγεςϜͷ(Ұ෦ͷ)ΞʔΩςΫνϟ Domain Detector weather rule-based NLU Global Pattern NLU other domain rule-based NLU1 weather ML-based NLU other domain ML-based NLU1 other domain rule-based NLU2 other domain ML-based NLU2 Global Pattern NLU • ࠷΋༏ઌ౓͕ߴ͍
 → ଞͷίϯϙʔωϯτͷӨڹड͚ͳ͍ • ओʹϏδωε্ॏཁͳυϝΠϯΛ
 ϧʔϧϕʔεͰਖ਼֬ʹ࣮૷ 21

NLUγεςϜͷ(Ұ෦ͷ)ΞʔΩςΫνϟ Domain Detector weather rule-based NLU Global Pattern NLU other domain rule-based NLU1 weather ML-based NLU other domain ML-based NLU1 other domain rule-based NLU2 other domain ML-based NLU2 Domain Detector • υϝΠϯ෼ྨ͕੹຿ • ൑ఆͨ͠υϝΠϯʹରԠ͢Δ
 domain-specific NLUʹॲཧΛ౉͢ • ଟΫϥε෼ྨʹͷΈूத͢Ε͹ྑ͍ 22

NLUγεςϜͷ(Ұ෦ͷ)ΞʔΩςΫνϟ Domain Detector weather rule-based NLU Global Pattern NLU other domain rule-based NLU1 weather ML-based NLU other domain ML-based NLU1 other domain rule-based NLU2 other domain ML-based NLU2 domain-specific NLU (DSNLU) • 1ͭͷυϝΠϯͷΈ୲౰ • DomainҎ֎ͷɺNLUͰऔಘ͢΂͖ ৘ใΛऔಘ͢Δͷ͕੹຿ • ଞͷDomainʹӨڹ͕ͳ͍ͷͰ
 มߋ͕༰қ 23

NLUγεςϜͷ(Ұ෦ͷ)ΞʔΩςΫνϟ Domain Detector weather rule-based NLU Global Pattern NLU other domain rule-based NLU1 weather ML-based NLU other domain ML-based NLU1 other domain rule-based NLU2 other domain ML-based NLU2 domain-specific rule-based NLU • ࡉ͔͍࢓༷΁ͷରԠΛߦ͏ • ϧʔϧϕʔεͳͷͰਖ਼֬ʹ
 ࢓༷௨Γʹ࣮૷͢Δ͜ͱ͕Մೳ 24

NLUγεςϜͷ(Ұ෦ͷ)ΞʔΩςΫνϟ Domain Detector weather rule-based NLU Global Pattern NLU other domain rule-based NLU1 weather ML-based NLU other domain ML-based NLU1 other domain rule-based NLU2 other domain ML-based NLU2 domain-specific ML-based NLU • rule-based NLUͰर͑ͳ͔ͬͨ
 ൃ࿩ʹରԠ • rule-based΄Ͳਖ਼֬Ͱ͸ͳ͍΋ͷͷ
 ޿ൣғͷൃ࿩ΛΧόʔͰ͖Δ 25

NLU ΞʔΩςΫνϟͷ·ͱΊ • NLUγεςϜΛ੹຿, Domain͝ͱʹ෼͚Δ͜ͱͰ • ෳࡶͳґଘؔ܎Λܰݮ • Domain಺ͷ࢓༷มߋʹૉૣ͘ରԠ͢Δ͜ͱ͕Մೳʹ • Rule-based, ML-basedͷ྆ํ࢖͏͜ͱͰ • ඍௐ੔ΛՄೳʹͭͭ͠ɺ޿ൣғͷൃ࿩ΛΧόʔ 26

ػցֶशϞσϧͷֶश, ഑৴, ӡ༻ Airflow, RekcurdΛ༻ֶ͍ͯश, ഑৴, ӡ༻աఔΛࣗಈԽ 27

ػցֶशϞσϧͷ؅ཧͷେ·͔ͳεςοϓ 28 σʔλूܭ σʔλͷϑΥʔϚοτ ֶश ഑৴ ධՁ ӡ༻

ػցֶशϞσϧͷ؅ཧͷࡉ͔͍εςοϓ • format data • train a new model • evaluate the model • share the result to members • Implement API to use model • estimate machine resource • set machine resource • Implement client side • deploy to development environment • set endpoint for dev API • confirm it works expectedly • deploy to staging environment • set endpoint for staging API • QA • minor fix • fix some training data • train a new model • deploy a fixed model to staging environment • set endpoint for staging API • deploy to production environment • blue-green deployment • set endpoint for production API • AB Test • monitoring • trouble shooting • rollback • … 29

ػցֶशϞσϧͷ؅ཧͷࣗಈԽ • ֶशաఔͷࣗಈԽ → Airflow • Ϟσϧͷ഑৴ͱධՁͷࣗಈԽ → Rekcurd from Airflow • σϓϩΠޙͷϞσϧͷӡ༻ → Rekcurd 30

What is Rekcurd ػցֶशϞδϡʔϧͷ഑৴, ؅ཧ, ӡ༻Λ؆୯ʹ͢ΔOSS https://github.com/rekcurd • ഑৴Λ؆୯ʹ → Rekcurd • ؅ཧͱӡ༻Λ؆୯ʹ → Rekcurd Dashboard (x Kubernetes) • طଘγεςϜ΁ͷ౷߹Λ؆୯ʹ → Rekcurd Client 31

Rekcurd pipͰinstallͯ͠rekcurdίϚϯυͰςϯϓϨʔτ࡞੒ޙɺ ҎԼͷؔ਺Λ࣮૷ • Ϟσϧͷϩʔυ • ਪ࿦ • ϞσϧͷੑೳධՁ ͋ͱ͸Dashboard͔ΒͷGUIૢ࡞Ͱ Kubernetes্ʹ͜ͷίʔυΛσϓϩΠ 32

Rekcurd Ϟσϧͷϩʔυ ਪ࿦ ϞσϧͷੑೳධՁ 33

Rekcurd Dashboard • RekcurdαʔϏεΛҰݩ؅ཧ • Ϟσϧͷόʔδϣχϯάɺ੾Γସ͑΋GUI͔Βͷૢ࡞ͰՄೳ • ։ൃ؀ڥ, ຊ൪؀ڥͷ੾Γ෼͚΋Մೳ 34

Why Rekcurd • ػցֶशϞδϡʔϧΛ؅ཧ͢ΔͨΊͷAPI͕͋Γ
 Airflow͔Βૢ࡞͢ΔPlugin͕͋Δ https://github.com/rekcurd/airflow-plugin → AirflowͷWorkflow಺ͰػցֶशϞσϧͷૢ࡞͕Մೳ • σϓϩΠɺϞσϧ੾Γସ͑ɺ࡟আͳͲ͕GUI͔Βߦ͑Δ. • σϓϩΠ͢Ε͹ҎԼͷػೳ͕ࣗಈͰ෇༩͞ΕΔ • readable endpoint • load balancer • auto scaling • ... 35

What is Airflow • Workflow؅ཧOSSπʔϧ • DAG (༗޲ඇ८ճάϥϑ) ͱݺ͹ΕΔάϥϑͰWorkflowΛఆٛ • DAGͷఆٛ͸PythonίʔυͰߦ͏ • GUIͰDAGͷՄࢹԽ, ࣮ߦ, ࣮ߦͷεέδϡʔϦϯά͕Մೳ • GUI͔ΒPythonίʔυ΍ϩά΋֬ೝͰ͖Δ 36

Why Airflow • طଘͷֶशͷͨΊͷεΫϦϓτ͕Python΍bashͩͬͨͷͰ AirflowͰWorkflowΛ࡞Γ΍͔ͬͨ͢ • PythonίʔυͰͷΈWorkflow͕ఆٛͰ͖ΔͷͰ
 όʔδϣϯ؅ཧָ͕ • ϩά΍࣮ߦঢ়گΛGUIͰ֬ೝͰ͖Δͷָ͕ • ಋೖ͕؆୯ 37

Example DAG 38 ֶश → ϞσϧΛRekcurd DashboardʹΞοϓϩʔυ → sandbox؀ڥͰ࢖͏ → (ฒྻͰ)ධՁ༻σʔλͷΞοϓϩʔυ → ϞσϧͷੑೳධՁ → ݁ՌΛग़ྗ → if ੑೳ͕ྑ͍: development؀ڥͰ࢖͏ (ಈ࡞֬ೝͷεςοϓʹਐΉͨΊ) → else: ϞσϧΛRekcurd Dashboard͔Β࡟আ

Video

ػցֶशϞσϧͷ؅ཧͷࡉ͔͍εςοϓ • format data • train a new model • evaluate the model • share the result to members • Implement API to use model • estimate machine resource • set machine resource • Implement client side • deploy to development environment • set endpoint for dev API • confirm it works expectedly • deploy to staging environment • set endpoint for staging API • QA • minor fix • fix some training data • train a new model • deploy a fixed model to staging environment • set endpoint for staging API • deploy to production environment • blue-green deployment • set endpoint for production API • AB Test • monitoring • trouble shooting • rollback • … 40

ػցֶशϞσϧͷ؅ཧͷࡉ͔͍εςοϓ • format data • train a new model • evaluate the model • share the result to members • Implement API to use model • estimate machine resource • set machine resource • Implement client side • deploy to development environment • set endpoint for dev API • confirm it works expectedly • deploy to staging environment • set endpoint for staging API • QA • minor fix • fix some training data • train a new model • deploy a fixed model to staging environment • set endpoint for staging API • deploy to production environment • blue-green deployment • set endpoint for production API • AB Test • monitoring • trouble shooting • rollback • … 41 ɹɹɹɹ: AirflowͱRekcurdͰ ࣗಈԽ, ؆қԽ Ͱ͖ͨ෦෼

ػցֶशϞσϧͷ؅ཧํ๏ͷ·ͱΊ • ػցֶशϞσϧͷֶशʹAirflowΛಋೖ • ϞσϧͷֶशΛ΄΅ࣗಈԽ • ֶशաఔ, ϩάͷՄࢹԽ • Airflow͔ΒRekcurdʹΞΫηε͢Δ͜ͱͰϞσϧͷ഑৴ΛࣗಈԽ • Ϟσϧ࡞੒ → Ξοϓϩʔυ&഑৴ΛϊϯετοϓͰ࣮ߦ • ϞσϧͷੑೳධՁ΋࣮ߦ • ػցֶशϞσϧͷӡ༻ʹRekcurdΛಋೖ • Ϟσϧͷ੾Γସ͑ͳͲͷ؅ཧΛGUI͔Βߦ͑Δ • k8s্ͰػցֶशͷαʔϏε͕HAߏ੒Ͱӡ༻͞ΕΔ 42