AUTOML IN CHATBOT BUILDER FRAMEWORK

JAEWON LEE
Data Scientist, NAVER Clova AI Business Chatbot Model

Presentation at LINE Thailand Developer Conference 2019

Transcript

1. JAEWON LEE Data Scientist NAVER Clova AI Business Chatbot Model AUTOML IN CHATBOT BUILDER FRAMEWORK
2. 
   NAVER Corp. Table of Contents 1. Chatbot ?? Chatbot !! 2. Clova Chatbot Builder Framework 3. What is in our Chatbot Builder Framework? 4. Why we can provide high performance Chatbot? 5. Conclusion
3. 
   NAVER Corp.
    1. Chatbot?? Chatbot!!
4. 
   NAVER Corp. Catch me, if you can… Show me the menu Any other options? I want a pizza Please select the option Please select the option Please select the option * The scenario has nothing to do with the actual products
5. 
   NAVER Corp. I know everything! Tiger Woods I meant the golfer, Tiger Woods! 27 men were killed by tiger during the last four years… Playing album, “Taste – Tyga”
6. 
   NAVER Corp. Let’s try ours X CLOVA Chatbot Builder
7. 
   NAVER Corp.
    2. Clova Chatbot Builder Framework, and its usage…
8. 
   NAVER Corp. Overview
   The global chatbot market is growing at an annual rate of 35%. Expecting $3.5 billion in 2021” (Technavio)
   By 2020, 55 percent of large companies will use more than one chatbot .. In 2021, more than 50 percent of companies will spend more on chatbot than mobile apps.”
   Chatbot is expected to help us save more than $2 billion this year and $8 billion annually by 2022 (Juniper Research)
9. 
   NAVER Corp. Overview ü Chatbot in one day ü Pre-built industry templates ü Less scenario, better performance Easy to BUILD ‘Flexible, Quick and Smart Chatbot’ Channels ü Leverage LINE platform ü Legacy DB I/F for Enterprise ü Format adjustment for messengers Easy to SERVE ü Supports 6 or more languages ü Compatible with speakers ü No limit on # of intents, scenario Easy to EXPAND Chatbot Builder with
   LINE Messenger Platform”
10. 
    NAVER Corp. Current Status AS - IS TO - BE Reference § Internal - 10+ Customer Service FAQ Chatbots § External (Enterprise Clients) - 60+ Chatbots (Finance, Telecom, Government, etc.) § Internal - 10+ Customer Service FAQ Chatbots § External (Enterprise Clients) - 300+ Chatbots (Covering all industries) - Global Coverage
11. 
    NAVER Corp. Use Case NAVER CS Chatbot (Internal) § Covers FAQ & Chitchat § 800K+ Chat requests § 140K+ Users § NAVER Community(Café), Blog, etc.
12. 
    NAVER Corp. Use Case CS Chatbot (Energy) Schedule Assistant (Healthcare) Recommendation (Retail)
13. 
    NAVER Corp. By combining STT/TTS Solution with our Clova Chatbot, connect Chatbot and Callbot within one framework. Moreover… OR Phone Call Mobile APP
     CLOVA PARTNER Customer Service Center Legacy(DB) CUSTOMER CHATLOG SOCIAL Customer Support Search CS Manual Sort Customer Info Search sample cases A A’ Internal CS MID BACK General CS (AI) Detail CS Input TEXT VOICE FRONT CANNOT RESPOND CONTACT CS AI-based Customer Service Process (Example of AI-ARS)
14. 
    NAVER Corp.
     Let’s see our Chatbot Builder
15. 
    NAVER Corp.
16. 
    NAVER Corp.
17. 
    NAVER Corp.
18. 
    NAVER Corp.
19. 
    NAVER Corp.
20. 
    NAVER Corp. You can do more… 1. Use External API to provide answers ${ActionMethod} Answer : Today’s weather in Bangkok is ${weather} Example : ${Weather} = http://weather.com?location=#{Bangkok} 2. Provide Mutiple-Choice Type or Short Answer Type ${Form} 3. Complicated customer order & Slot filling to complete such actions from each customer ${Task} $ # We provide various functions with conversational components
21. 
    NAVER Corp.
22. 
    NAVER Corp.
23. 
    NAVER Corp.
24. 
    NAVER Corp. You can do more… and more… We provide not only text messages but text balloons from each messenger platform We provide basic components such as Text, Button, Image Type, Flex Message, Carousel, Quick Reply etc
    
25. 
    NAVER Corp. You can do more… and more…
26. 
    NAVER Corp.
     3. What is in our Chatbot Builder Framework?
27. 
    NAVER Corp. Chatbot Builder Framework Process Overview Build —> Train —> Tune —> Deploy 1st Stage:
    User QA Dataset Preprocessing [Hadoop] 2nd Stage:
    [GPU] Prepare for training
    [Tensorflow or Various ML Framework] 3nd Stage:
    Model tuning and deployment [Server]
28. 
    NAVER Corp. Insert Scenario to Builder § Create Domain § Use our platform to insert dialogue scenario § Various data generation tools such as Action-method, Form, State, Answer Tagging can enhance each dialogue based on user preference
29. 
    NAVER Corp. Pre-process Human Languages § Baloo § Pre-process data scenario § Extract language features § Request/Manage all model builds from c3dl GPU § Rama § Support multi-lingual tokenizers § Part-of-Speech Tagging § EOMIS Tagging(for Korean only) § Named Entity Tagging
30. 
    NAVER Corp. Train Models § Train models with pre-processed data corpus § Akella: RNN models § Jacala: Classifier models § Raksha: Embedding models § Bagheera: Multi-turn detector
31. 
    NAVER Corp. Sequence Model Learn various sentence and grammar: Vector Representation / N-hot Representation Memorize important information: LSTM Stacked Study sentence in forward & backward: Bi-directional Review sentence again: Residual / Highway Network Feedbacks based on answers: Attention Find the answer based on various opinions : Model Ensemble
32. 
    NAVER Corp. Semantic Model DSSM – Compute Similarity in Semantic Space xt Word sequence Relevance measured by cosine similarity f1 , f2 , …, fTQ w1,w2, …,wTQ 128 sim(X, Y) f1 , f2 , …, fTD1 w1,w2, …,wTD 128 X Y g ( . ) f ( . ) Multi-Layer Perceptron (MLP) if text is a bag of words [Huang+ 13]
33. 
    NAVER Corp. Embedding Model Pre-trained FastText Vector Train Query Train Query Feature Vector Tokenizing Answer Test Query Test Query Feature Vector Averaging cos - similarity Train & Prediction Threshold • Wikipedia • Chat Corpus • Twitter § Compute similarity between trained query and input query to provide pre-paired answer § Improve response coverage since global embedding can cover where local embedding cannot cover
34. 
    NAVER Corp. Classifier Model Embedding Layer, Wini § Currently use global embedding vectors such as fastText § Apply various pre-trained embedding vectors such as Elmo, BERT etc Classification Layer, Baloo & Kaa § When training, fine-tune only the top of layer § When serving, deploy only middle part of network as inference § Apply CNN, FeedForward NN, Transformer in progress
35. 
    NAVER Corp. ) 2 ) 1 )2 A ) ) $ 2 12 2 ) 2 ) 2 ) 2 2 In short
36. 
    NAVER Corp. Tune Models § Cluster given scenario data to better train deep learning models § Find the best domain-specific hyperparameters for each model § Find the best ensemble weights for each model
37. 
    NAVER Corp. Serve Chatbot Engine § Provide the most appropriate ensemble answer to one specific platform § Serve multiple messenger platforms such as LINE, facebook messenger, etc. § Use Akka Sharding to provide non-stop service by communicating with each server
38. 
    NAVER Corp.
     4. Why we can provide high performance Chatbot?
39. 
    NAVER Corp. AI learn languages by… § Vector Representation § N-hot Embedding Vectors from given data corpus § Global Embedding Vectors from Glove, fastText, TAPI, etc. § Locate all words/sentence properly within given vector space § N elements in N-dimensions represent characteristics of each sentence § The Closer embedding vectors, The More Semantically matching
40. 
    NAVER Corp. I can learn by myself?! Embedding vectors are for me to understand languages. Give me some time... Then I can do better and I can do by myself !!!
41. 
    NAVER Corp. Now Let’s Apply AutoML § The process of automated end-to-end process by applying machine learning algorithms § AI-based solution to the ever-growing challenge by applying machine learning algorithms § Data Preprocessing, Algorithm Selection, Hyperparameter Optimization, etc.
42. 
    NAVER Corp. Here’s what we have Model Selection Data Preprocessing Data Corpus Features Extraction Features Selection Features Construction Parameter Optimization Model Validation § Iterate and Train multiple times to optimize and find local minimum for each data corpus
43. 
    NAVER Corp. AutoML Pipeline in detail Q & A Scenario Candidate Answer Model Ensemble Query Pre-Processing Feature Extraction Global Vector Representation Embedding Model Sequence Model Semantic Model N-hot features Clustering Model Tuning POS EOMI Named Entity
44. 
    NAVER Corp. AutoML Pipeline in detail
45. 
    NAVER Corp. Validation Set? Validation Score? § Determine whether each query is located at the proper vector space § Use Cross-Validation to split train and valid set § Randomly sample from dataset with the ratio of 8:2 § Run several times to prevent model to be less biased § Or, Remove 1~2 queries from each group for evaluation § For each group, choose queries far from the center by computing cosine similarity
46. 
    NAVER Corp. Auto-Evaluation in Clustering Q & A Scenario Clustering Train s2s Model MSE Auto-Evaluation § Create validation set after every clustering process § Run semi-infinite clustering process until it reaches to local minimum § Choose the best “k” with the best MSE score
47. 
    NAVER Corp. Auto-Evaluation in Hyperparameter Tuning § Suppose each clusters are well-organized by models § More Efficiently? More Accurately? § Coordinate Search, Random Search, Genetic Algorithms § Bayesian Hyper-parameter Optimization § Use Cosine Similarity, MSE or Hit Ratio for validation score
48. 
    NAVER Corp. Auto-Evaluation in Hyperparameter Tuning Q & A Scenario Clustering Compute Validation Score hiddenSizes #1 hiddenSizes #2 hiddenSizes #3 hiddenSizes #4 hiddenSizes #5 hiddenSizes #6 hiddenSizes #7 hiddenSizes #8 hiddenSizes #9 § Start from the most effective hyperparameter such as encoder hidden sizes
49. 
    NAVER Corp. Auto-Evaluation in Ensemble Weights § Assume there exists a model that answers better than the others § For the given query, Response Selector provides final answer that maximizes the linear combination of each model’s prediction intensity such as cosine similarity and the accuracy of each model as the selection strength so-called ensemble weights
50. 
    NAVER Corp. Auto-Evaluation in Ensemble Weights Query Model 1 (! 1 ) Model 2 (! 2 ) Model 3 (! 3 ) ⋮ Answer 1 Answer 2 ⋯ cos-sim1, 1 cos-sim2, 1 cos-sim3, 1 cos-sim1, 2 cos-sim2, 2 cos-sim3, 2 Answer 3 cos-sim1, 3 cos-sim2, 3 cos-sim3, 3 Chatbot Weighted Avg cos-sim1 cos-sim2 cos-sim3 ⋮ ⋮ ⋮ ⋯ ⋯ ⋯ ⋯ ⋱ Take answer of the index Max Avg § Coordinate Search to find optimized weights for each model
51. 
    NAVER Corp.
     5. Conclusion
52. 
    NAVER Corp. WHY CLOVA? Build a AI Chatbot that truly understands what people say, not just their intents and keywords It is difficult to cover all combinations of human expression just by adding simple question and answers. CLOVA chatbot not only understands incoming queries, but also chooses where to get the best answer WHY NAVER? WHY CLOVA? ü Anyone could build and deploy chatbot ü Built-in dashboard to control different domains ü Visualize and re-train chatbot through chat history and statistics
53. 
    NAVER Corp. Chatbot - Benchmark Performance(Hit ratio) Methods ü Test under same condition for each chatbot builder ü Used Japanese dataset and translated into Korean and English ü Randomly selected test queries from the sample data - Korean: train(1,665), test(159) - English: train(512), test(152) 95.6% 91.8% 88.7% 72.3% KOREAN 91.4% 90.1% 88.2% ENGLISH 91.4% “Utilizing volume of data collected from search engine, CLOVA has great Korean language processing capabilities, and through auto-tune features it is able to learn new languages with less amount of data”
54. 
    NAVER Corp. → Cooperation with Clova, NBP, LINE + Why Clova Chatbot Builder? Provide high quality conversation system with minimum labor cost → B2B Template & Strong NLP Engine Support multilingual conversation → Minimum language barrier NLP Engine Continuous update with users’ feedback Successfully launched Clova Chatbot Builder Framework, To be positioned as an “AI Messenger Platform” Conclusion
55. 
    NAVER Corp.
     Questions ?
56. 
    NAVER Corp. Appendix jaewonlee.728@navercorp.com jp24147@linecorp.com Chatbot Builder Quick Start Guide http://docs.ncloud.com/en/chatbot/chatbot-1-1.html
57. THANK_YOU