ML & Chatbots workshop - Speaker Deck

Slide 1

Slide 1 text

Nederlandse Spoorwegen Machine Learning Workshop Tessa Reef Lee Boonstra 8 July 2019

Slide 2

Slide 2 text

▶ 11.00 - 11.30 Arrival/coffee & Intro by Tessa ▶ 11.30 - 12.30 Chatbots & Google Assistant by Lee ▶ 12.30 - 13.00 Lunch ▶ 13.00 - 14.00 Cloud AI by Lee ▶ 14.00 - 14.15 Quiz - (Win a Google Home!) ▶ 14.15 - 14.30 Conclusions by Tessa Agenda

Slide 3

Slide 3 text

Lee Boonstra @ Google Customer Engineer, Google Cloud (2,5 years) Dialogflow Expert Chapter Lead ERG Gayglers Public Speaker (since 2013) Writer/Blogger for Techzine, .Net Magazine, Marketingfacts.nl, CustomerTalk.nl and Google Cloud Blog www.leeboonstra.com

Slide 4

Slide 4 text

Lee Boonstra before Google Book Writer O’Reilly (Hands-on Sencha Touch 2, mobile web development) Technical Trainer @ Sencha Inc. 2012 - 2016 Lead Client-side Engineer @ Valtech 2009 - 2012 Senior Java Developer @ Accenture 2007 - 2009 Founder of my own company 2004 www.leeboonstra.com

Slide 5

Slide 5 text

Improve your customer care by building an AI platform with the use of Google Cloud Lee Boonstra Sales engineer Google Cloud

Slide 6

Slide 6 text

Chatbots are expected to trim business cost by more than $8 billion per year by 2022 Juniper Research

Slide 7

Slide 7 text

Chatbots exists since the 90’s… So why are they popular now?

Slide 8

Slide 8 text

FIRST CHATBOT, 1994

Slide 9

Slide 9 text

SHIFT FROM A MOBILE FIRST TO AN AI FIRST STRATEGY

Slide 10

Slide 10 text

HOW DID YOU LEARN YOUR FIRST LANGUAGE?

Slide 11

Slide 11 text

11 Chatbots is all about Machine Learning! ● Natural Language Understanding ● Intent Matching ● Speech to Text ● Text to Speech (Wavenet)

Slide 12

Slide 12 text

How to create chatbots?

Slide 13

Slide 13 text

No content

Slide 14

Slide 14 text

GOOGLE CLOUD HAS OVER 100 BUILDING BLOCKS INSTEAD LET’S FOCUS ON SOLUTIONS

Slide 15

Slide 15 text

Introducing Cloud AI Less ML expertise More ML expertise Cloud AI solutions Cloud Job Discovery Contact Center Document Understanding Cloud AI building blocks Cloud AI platform Cloud ML Engine Cloud Video Intelligence ML professionals & service partners ASL Professional Services Organization Cloud AutoML Vision Vision Cloud Vision Language Cloud Natural Language Cloud AutoML NL Dialogflow Enterprise Cloud Translation Cloud Speech-to-Text Cloud Text-to-Speech Cloud AutoML Translation New New New Cloud GPU Cloud TPU Cloud Dataflow Cloud Dataproc Machine & Deep Learning ML accelerators ML libraries Tensorflow Kubeflow Kaggle/datasets Datasets

Slide 16

Slide 16 text

16 ● Previously known as API.AI ○ (Sept 2016, acquired by Google) ● Powered by Machine Learning: ○ Natural Language Understanding (NLU) ○ Intent Matching ○ Conversation Training ● Cross platform ● Build faster with the Web UI ● Scalable: separate your conversation text from code ● Speech / Voice Integration ● Multi-lingual bot support (20+ languages) ● Direct integration with 15+ channels like Google Assistant, Slack, Twilio, Facebook... Development suite for building Conversational UIs.

Slide 17

Slide 17 text

Dialogflow for Enterprises ● Built on Google Cloud Platform infrastructure, easy integration with over 100 Cloud components ● Cloud Support and SLA available ● Compliance ● Extensive Documentation and training programs available. ● Powerful IAM; User Roles and Permissions ● Stackdriver integration for automatic logging, debugging, tracing and error reporting ● Unlimited API call quotas

Slide 18

Slide 18 text

Types of Chatbots and the use cases

Slide 19

Slide 19 text

Three types of chatbots Chatbots Chatbots in websites, apps, or on social media like Facebook Messenger, Slack.... Voice Activated Speakers Smart Assistants, like Google Assistant, Alexa, Siri, on mobile phones and devices like Google Home, Google Hub, Watches, TVs... Callbots Chatbots integrated in IVR systems, phone reservation systems, contact centers...

Slide 20

Slide 20 text

Chatbot / Web Chatbot ● Chatbots for internal processes. ● Chatbots that face customers. ● Chatbots to collect feedback. ● Topic Modelling ● Chatbots for intent matching (Natural Language for Searching on websites.) ● Chatbots on social media.

Slide 21

Slide 21 text

Use Case: ING Bank Public facing chatbot Inge of ING Bank. Customers can ask ‘Inge’ information about their accounts and debit cards. Inge can detect the sentiment. When customers get frustrated, it will automatically route the user to a human agent.

Slide 22

Slide 22 text

Architecture: Chatbots Dialogﬂow Enterprise Website Human Agent

Slide 23

Slide 23 text

Google Assistant Action ● Voice interface will be the future, since it’s very accessible. ● Google Assistant has over 1 billion of users. ● According to Adobe Analytics, 71% of owners of smart speakers use voice assistants at least daily, and 44% using them multiple times a day. ● Extend the Google Assistant with your apps. Users expect your brand to be available on smart speakers.

Slide 24

Slide 24 text

Use Case: Rabobank Ok Google, talk to Rabobank. The Rabobank Assistant can help you with banking via voice. You can request your balance, transfer money or set budget notifications. It’s available in multiple languages for Google Assistant on mobile devices and on the Google Home.

Slide 25

Slide 25 text

Conﬁdential + Proprietary Conﬁdential + Proprietary You will need to design your conversation

Slide 26

Slide 26 text

26 Website Filter on account name or account number Lot’s of results on a screen.

Slide 27

Slide 27 text

27 Website with Natural Language... Natural way of asking!

Slide 28

Slide 28 text

28 How much have I spent on taxis last month? It looks like, you spent about 20 euros on taxis last month. You took the TCA taxi twice. Voice channels There’s no screen. Focus on the conversation.

Slide 29

Slide 29 text

29 It looks like, you spent about 20 euros on taxis last month. You took the TCA taxi twice. Here’s an overview: Voice channels with screens How much have I spent on taxis last month? Focus on the conversation. But also display stuff.

Slide 30

Slide 30 text

30 Assistance is not just about voice

Slide 31

Slide 31 text

Conﬁdential + Proprietary Conﬁdential + Proprietary What happens under the hood?

Slide 32

Slide 32 text

Google | Proprietary & Confidential 32 GOOGLE ASSISTANT USER Hey Google.. Idle

Slide 33

Slide 33 text

Google | Proprietary & Confidential 33 GOOGLE ASSISTANT USER Hey Google.. ..will it rain today? Listening

Slide 34

Slide 34 text

Google | Proprietary & Confidential 34 GOOGLE ASSISTANT USER WEB SERVER Hey Google.. ..will it rain today? GET www.weather.com/info city: Amsterdam Date: 2019-02-06 Recognizing

Slide 35

Slide 35 text

Google | Proprietary & Confidential 35 GOOGLE ASSISTANT USER WEB SERVER Hey Google.. ..will it rain today? GET www.weather.com/info city: Amsterdam Date: 2019-02-06 { location: “amsterdam” weather: “rain”, temperature: 8 } Thinking

Slide 36

Slide 36 text

Google | Proprietary & Confidential 36 GOOGLE ASSISTANT USER WEB SERVER Hey Google.. ..will it rain today? GET www.weather.com/info city: Amsterdam Date: 2019-02-06 { location: “amsterdam” weather: “rain”, temperature: 8 } Yes, it will rain in Amsterdam all day today. Speaking

Slide 37

Slide 37 text

Conﬁdential + Proprietary Conﬁdential + Proprietary How can you build your own action on top of the Google Assistant?

Slide 38

Slide 38 text

38 3rd party integration Extend the Google Assistant with your own custom actions. Hey Google, let me talk to BookAMeetingRoom Welcome, how can I help you? I want to book a meeting room for 3 persons. Let’s get BookAMeetingRoom Sure, for when? Tomorrow, from 2pm to 3pm.

Slide 39

Slide 39 text

39 Ok Google, talk to __________. Ok Google, connect me with __________. Ok Google, get __________. Start a 3rd party action There is an app directory! (appstore). And the Google Assistant can give app suggestions.

Slide 40

Slide 40 text

40 ● Write the conversation - Dialogflow (Enterprise) ● Deploy on GA+ UX components - Actions on Google Optional: ● Back-end integration - Fulfillment app (dialogflow/aog SDK) ● Communication to back-ends - Your own APIs What do I need to build my own action?

Slide 41

Slide 41 text

41 ● Bring your agents to smart speakers (Google Home) or phones (Android, iOS app) ● Actions on Google includes: ○ UI toolkit, ○ Audio toolkit ○ Account Linking API ○ SDKs ○ tutorial guides ● UI components such as: ○ Buttons, Images ○ Cards, Carousels, ○ Lists ○ Tables Program for developers of Actions (“apps”) that run via Google Assistant Actions on Google

Slide 42

Slide 42 text

Architecture: Google Assistant Dialogﬂow Enterprise Google Assistant

Slide 43

Slide 43 text

Contact Center Frustrations ● Long waiting / hold times ● Unlimited Call transfers ● IVR difficult to navigate ● Availability ● Inadequate information ● Agents have to answer same types of questions over and over again.

Slide 44

Slide 44 text

With AI in your Contact Center Bots that listen and give on screen solutions to the human agent. ● Always answers the right question. ● Shorten hold times ● Shorten the call time Bots that understand your question. ● No longer you need to listen to audio recordings & press keys. ● You don’t need to be transferred from one agent to the other Bots that can answer / resolve common questions. ● Shorten hold times ● Shorten the call time ● Availability ● No longer you’ve been told to look on the website

Slide 45

Slide 45 text

Use Case: Health Insurance The (health) insurance sector deals with contact center spikes. At the end of the year, customers are able to change their insurance. Which results in long waiting times and students that aren’t trained, picking up the phone. Calls needs to be monitored, to gather analytics about the type of questions and provided service.

Slide 46

Slide 46 text

Architecture: Contact Center without AI Call Center Agent

Slide 47

Slide 47 text

Architecture: Contact Center Dialogﬂow Enterprise Agent Assist Call Center Agent Text to S Speech

Slide 48

Slide 48 text

Demo’s

Slide 49

Slide 49 text

Babs the Banking Bot Web Chat Google Assistant Hey Google, let me talk to Babs The Banking Bot Welcome, how can I help you? I want to transfer money. Let’s get Babs the Banking Bot How much do you want to transfer? 100 euro.

Slide 50

Slide 50 text

Which customers are unhappy and why? (Analytics)

Slide 51

Slide 51 text

How can I improve the user experience? (Analytics)

Slide 52

Slide 52 text

Collect real-time chats from Dialogﬂow SDK

Slide 53

Slide 53 text

Mask sensitive Information with DLP API

Slide 54

Slide 54 text

Understand the text with NLP API

Slide 55

Slide 55 text

Store all data in a data-warehouse

Slide 56

Slide 56 text

Optimize your agent

Slide 57

Slide 57 text

Confidential + Proprietary Advanced Chatflow with machine learning bot analytics User types to custom UI or channel Chatbot replies Dialogflow Enterprise Customer Client JS Angular 5 web front-end Kubernetes Engine Chat Server Dialogflow SDK / socket.io Kubernetes Engine Back-end CRM Python / Django Kubernetes Engine Container Registry Containers images can be stored in the Container Registry Messaging Publisher Pub/Sub Webhook Router Cloud Function Webhook Container Builder Building Dev Pipelines

Slide 58

Slide 58 text

Confidential + Proprietary Advanced Chatflow with machine learning and bot analytics User types to custom UI or channel Chatbot replies Dialogflow Enterprise Customer Client JS Angular 5 web front-end Kubernetes Engine Chat Server Dialogflow SDK / socket.io Kubernetes Engine Back-end CRM Python / Django Kubernetes Engine Subscription Cloud Function Sensitivity Filter DLP API Sentiment Detector NLP API Data Warehouse BigQuery Messaging Publisher Pub/Sub Webhook Router Cloud Function Webhook

Slide 59

Slide 59 text

Cloud AI: Make your workloads smarter Lee Boonstra Sales engineer Google Cloud

Slide 60

Slide 60 text

Why now? ● Amount of data Better Models More Computing Power

Slide 61

Slide 61 text

Machine Learning to classify things Dog vs. Mop Oh, that’s easy...

Slide 62

Slide 62 text

Wait what?!

Slide 63

Slide 63 text

We would need machine learning to give us results Confidence level

Slide 64

Slide 64 text

It’s inspired by how our brains work Neural Networks

Slide 65

Slide 65 text

It’s easier to make computers learn than to build smarter computers

Slide 66

Slide 66 text

Why Google? Data Scientists use Tensorflow Tensorflow is what we use for our own internal machine learning projects, and now it’s available to you! Google made it open source. ● More than 480 contributions ● 10,000 commits in a year ● 53k star rating http://www.tensorflow.org

Slide 67

Slide 67 text

Google is an AI company Used across products: Unique project directories Time

Slide 68

Slide 68 text

Conﬁdential + Proprietary Conﬁdential + Proprietary What’s the state of the industry?

Slide 69

Slide 69 text

Very few people can create custom ML models today Who can actually use AI today? 10K DL researchers 2M ML experts +23M Developers +100M Business users

Slide 70

Slide 70 text

Compute is Critical 80% of recent AI advances can be attributed to more available computing power Dileep George, Cofounder of the Machine Learning Startup Vicarious

Slide 71

Slide 71 text

CLOUD AI PLATFORM’S GOAL Enable generalist software engineers to easily build and run custom AI applications anywhere.

Slide 72

Slide 72 text

How do you go from data to intelligent actions?

Slide 73

Slide 73 text

Three ways to go from data to intelligent actions Train Custom ML Model ● Need a custom ML model ● Have a team of data scientist ● Run Hybrid / On-premise Pretrained Google ML Models ● Don’t have much data ● Have a team of developers ● Run as full AI solution (no developers or data scientists needed) Retrain a Google ML Model ● Need a custom ML model ● Have a team of developers

Slide 74

Slide 74 text

Slide 75

Slide 75 text

Train a custom machine learning model Support for custom ML Models Cloud AI Platform Hardware optimised for your problem Cloud TPU, GPU, CPU Any ML framework Cloud ML Engine Managed Portable Portable & Open Kubeflow One stop AI catalog AI Hub

Slide 76

Slide 76 text

Slide 77

Slide 77 text

ML Engine - Managed Machine Learning End to End Machine Learning pipeline Data ingestion 1 Data analysis 2 Data transformation 3 Train 4 Model evaluation 5 Model validation 6 Deploy 7 ● Managed service to make training & prediction easy ● Easy distributed training ● Hyperparameter tuning ● Top 4 frameworks ● Custom container support coming soon

Slide 78

Slide 78 text

Kubeflow Run portable & scalable ML workloads on Open Source Kubernetes Easy to get started ● Out-of-box support for top frameworks ○ pytorch, caffe, tf and xgboost ● Kubernetes manages dependencies, resources Swappable & scalable ● Library of ML services ● CPU, GPU, TPU ● Massive scale Meet customer where they are ● GCP ● On-prem ML microservices Kubernetes Cloud On-prem Training Predict Training Predict … …

Slide 79

Slide 79 text

AI Hub One stop AI catalog Easily discover plug & play pipelines & other content built by Google AI. 01 Private hosting Host pipelines and ML content with private sharing controls within an enterprise to foster reuse within organizations. 02 Easy deployment on GCP and hybrid Deploy pipelines via Kubeﬂow on GCP and on premise. 03 (g.co/aihub)

Slide 80

Slide 80 text

What is included? Kubeflow (On premises) AI Platform Integrated with Deep Learning VM Images Cloud Dataﬂow Cloud Dataproc Google BigQuery Cloud Dataprep Google Data Studio Notebooks Data Labeling Training Predictions Pre-built Algorithms For data warehousing For data transformation For data cleansing For Hadoop and Spark clusters For BI dashboards AI Hub

Slide 81

Slide 81 text

Cloud Datalab OSS Jupyter notebook GCP integrations (BigQuery, Cloud Storage, etc) Run locally or powered by GCE

Slide 82

Slide 82 text

Cloud TPU Offering Cloud TPU v2 180 teraflops 64 GB HBM training and inference Cloud TPU v2 PodALPHA 11.5 petaflops 4 TB HBM 2-D toroidal mesh network training and inference Cloud TPU v3ALPHA 420 teraflops 128 GB HBM training and inference

Slide 83

Slide 83 text

No content

Slide 84

Slide 84 text

Use Pre-trained Google machine learning model Sight Cloud Vision Cloud Video Intelligence Language Cloud Translation Cloud Natural Language Conversation Cloud Speech-to-Text Dialogﬂow Enterprise Edition Cloud Text-to-Speech

Slide 85

Slide 85 text

Speech API (STT) Powered by deep learning neural networking to power your applications.. No need for signal processing or noise cancellation before calling API. Can handle noisy audio from a variety of environments. Noise Robustness Can provide context hints for improved accuracy. Especially useful for device and app use cases. Word Hints Speech Recognition Recognizes over 80 languages & variants. Can also ﬁlter inappropriate content in text results Over 80 languages Can stream text results, returning partial recognition results as they become available. Can also be run on buffered or archived audio ﬁles. Real-time results

Slide 86

Slide 86 text

API Usage: Understand Speech - Batch Stored Audio Recognized text Speech API Create JSON request with the audio ﬁle and language of audio (default is en_US) Process the JSON response Call the REST API 1 2 3

Slide 87

Slide 87 text

API Usage: Understand Speech - Streaming Streaming Audio Speech API gRPC Recognized Text gRPC streaming request with initial context Real time streaming results while speaking Bi-directional: Streams audio in while stream text out 1 2 3

Slide 88

Slide 88 text

88 ● Native audio responses, complementing existing Speech-to-Text capability. ● Uses DeepMind’s WaveNet technology (It closes the voice-quality gap with human voice (based on the Mean Opinion Score for voice quality) by over 70 percent.) ● Also used for Phone Calls, when making use of the Phone Gateway. ● Device Profiles (Shape the waveform differently, depending on the speaker you use.) ● You will need to enable Automatic Text to Speech in Settings/Speech menu. Text to Speech (TTS)

Slide 89

Slide 89 text

89 ● Deep generative model of raw audio waveforms ● Voices sound natural and unique ● Capture subtleties like pitch, pace, and all the pauses that convey meaning ● Create New voices in weeks i.s. Months https://deepmind.com/blog/wavenet-generative-model-raw-audio/ DeepMinds WaveNet Technology

Slide 90

Slide 90 text

Natural Language API Identify entities and label by types such as person, organization, location, events, products and media. Enables you to easily analyze text in multiple languages including English, Spanish and Japanese. Extract tokens and sentences, identify parts of speech (PoS) and create dependency parse trees for each sentence. Syntax analysis Entity Recognition Multi-Language Support Understand the overall sentiment expressed in a block of text. Sentiment Analysis

Slide 91

Slide 91 text

DLP API (Data Loss Prevention) ● Mask, redact, and generalize PII and sensitive data with Machine Learning ● Scan for and anonymize sensitive data to comply with regulations or policies (text, text on ﬁle system, Cloud Storage, DataStore, BigQuery) ● Clear reporting for review and auditing

Slide 92

Slide 92 text

Translation API 100+ languages: from Afrikaans to Zulu. Automatically identify languages wiht high accuracy. Easy to use Google REST API. You don’t have to extract text from you document. Just send it HTML documents and get back translated text. Can seamlessly scale with almost any volume. If a higher quota is needed, you can simply request an increase. Highly scalable The Premium edition is tailored for users who need precise, long-form translation services (e.g. livestream translations, high volume of emails, detailed articles and documents) Premium edition BETA Detect + translate Simple integration TRY THE API

Slide 93

Slide 93 text

Vision API Detect broad sets of categories within an image, ranging from modes of transportation to animals. Analyze facial features to detect emotions: joy, sorrow, anger. Detect logos. Detect and extract text within an image, with support for a broad range of languages, along with support for automatic language identiﬁcation. Extract text Detect different types of inappropriate content from adult to violent content. Powered by Google Safe Search Detect inappropriate content Object Recognition Facial sentiment & logos

Slide 94

Slide 94 text

Video Intelligence API Detect entities within the video, such as "dog", "ﬂower" or "car". You can now search your video catalog the same way you search text documents.. Extract actionable insights from video ﬁles without requiring any machine learning or computer vision knowledge. Enable Video Search Label Detection Insights From Videos

Slide 95

Slide 95 text

Demo’s

Slide 96

Slide 96 text

Use Pre-trained Google machine learning model Sight AutoML Vision AutoML Video Intelligence Language AutoML Natural Language AutoML Translation Structured Data AutoML Tables

Slide 97

Slide 97 text

Retrain a Google machine learning model Cloud AutoML Dataset Train Deploy Serve Generate predictions with a REST API

Slide 98

Slide 98 text

Auto ML Custom task Generic task Someone else has solved this before Trained on common classes Specific to your dataset ML APIs TensorFlow “cat” “bob” AUTO ML Developer Data Scientist Developer or Data scientist

Slide 99

Slide 99 text

UPDATE DEPLOY EVALUATE TUNE ML MODEL PARAMETERS ML MODEL DESIGN DATA PREPROCESSING Introducing Cloud AutoML A technology that can automatically create a Machine Learning Model UPDATE DEPLOY EVALUATE TUNE ML MODEL PARAMETERS ML MODEL DESIGN DATA PREPROCESSING

Slide 100

Slide 100 text

Via a web interface you will go through these steps 1 3 4 5 Upload Labeled Data Evaluate Predict Iterate (if needed) 2 Train

Slide 101

Slide 101 text

Optimize the pre-trained models with your own data Auto ML ● Create models for your own domain, but use the Google pre-trained models as a base. ○ Vision, Translation, or NLP ● Web Interface, to upload a CSV with labeled data. ● Your use-case is: ○ Not supported by pre-built APIs AND ○ Has sufficient labeled training data ● You want to get to produce a model and predict quickly ● You don’t want to build a model from scratch

Slide 102

Slide 102 text

Demo’s

Slide 103

Slide 103 text

Conclusion

Slide 104

Slide 104 text

Conﬁdential + Proprietary Wouldn’t it be nice to build one AI solution that can answer all questions and is available from anywhere

Slide 105

Slide 105 text

BigQuery Dialogﬂow Enterprise Text to S Speech Google Assistant Website Social media Channel Agent Assist Call Center Agent Your System Social media Channel

Slide 106

Slide 106 text

Conﬁdential + Proprietary But if even if you just add one new AI channel. You can improve your customer experience and trim business costs.

Slide 107

Slide 107 text

Like smart assistants... Dialogﬂow Enterprise Google Assistant Call Center Agent

Slide 108

Slide 108 text

Thank you! My Examples http://www.futurebank.nl https://github.com/savelee/kube-django-ng https://github.com/GoogleCloudPlatform/tulip My Blue print https://cloud.google.com/blog/products/ai-machine-learnin g/simple-blueprint-for-building-ai-powered-customer-servi ce-on-gcp