Azure AI – Build 2020 Updates SATO Naoki (Neo) Senior Software Engineer Microsoft

No content

Fueled by breakthrough research 2016 Object recognition human parity 2017 Speech recognition human parity 2018 Reading comprehension human parity 2018 Machine translation human parity 2018 Speech synthesis near-human parity 2019 General Language Understanding human parity 2020 Document summary at human parity

1.8M Hours of meetings transcribed in real-time 1B PowerPoint Designer slides used Tested at scale in Microsoft solutions 80M Personalized experiences delivered daily Machine translation human parity Object detection human parity Switchboard Switchboard cellular Meeting speech IBM Switchboard Broadcast speech Speech recognition human parity Conversational Q&A human parity First FPGA deployed in a datacenter

Power Platform Power BI Power Apps Power Automate Power Virtual Agents Azure 58 Regions 90+ Compliance Offerings $1B Security investment per year 95% Fortune 500 use Azure Azure AI ML platform Customizable models Vision, Speech, Language, Decision Scenario-specific services Cognitive Services Azure Machine Learning Data Platform App Dev Platform & tools Compute Cognitive Search Bot Service Form Recognizer Video Indexer

Cloud CPU, GPU, FPGA Datasets Profiling, Drift, Labeling Inferencing Batch, Realtime MLOps Reproducible, Automatable, GitHub, CLI, REST Experience SDK, Notebooks, Drag-n-drop, Wizard Edge CPU, GPU, NPU IoT Edge Security, Mgmt, Deployment Compute Jobs, Clusters, Instances Model Registry Models, Images Training Experiments, Runs

Responsible AI

Practices Principles Tools AETHER committee The Partnership on AI Guidelines for Human-AI Design Homomorphic Encryption Interpret ML Differential Privacy Data Drift Secure MPC Guidelines for Conversational AI Fairness Reliability Privacy Inclusivity Accountability Transparency

Why responsible AI? Report

Understand Interpretability Fairness Azure Machine Learning Responsible ML Control Audit trail Datasheets Protect Differential privacy Confidential machine learning

Understand Interpretability Fairness Azure Machine Learning Responsible ML Control Audit trail Datasheets Protect Differential privacy Confidential machine learning

Loan Application Decisions Azure Machine Learning How does it decide who to accept or reject? Is my model fair? Create a model for loan application acceptance

Loan Application Decisions Azure Machine Learning How does it decide who to accept or reject? Is my model fair? Create a model for loan application acceptance

Fairness in AI There are many ways that an AI system can behave unfairly A voice recognition system might fail to work as well for women as it does for men A model for screening loan or job application might be much better at picking good candidates among white men than among other groups Avoiding negative outcomes of AI systems for different groups of people

Assessing unfairness in your model https://github.com/fairlearn/fairlearn Fairness assessment: Usecommonfairness metrics andaninteractive dashboardto assess which groups of peoplemay benegatively impacted Model formats: Python models using scikit predict convention, Scikit, Tensorflow, Pytorch, Keras Metrics: 15+ common group fairness metrics Model types: Classification, Regression Fairness mitigation: Use state-of-the-art algorithms to mitigate unfairness in your classificationandregressionmodels

Fairness Assessment Input Selections Sensitive attribute Performance metric Assessment Results Disparity in performance Disparity in predictions

Fairness Mitigation Fairness Criteria Demographic Parity Equalized Odds Mitigation Algorithm Postprocessing Method Reduction Method

Loan Application Decisions Azure Machine Learning How does it decide who to accept or reject? Is my model fair? Create a model for loan application acceptance

Understand and debug your model Interpret Glassbox and blackbox interpretability methods for tabular data Interpret- community Additional interpretability techniques for tabular data Interpret-text Interpretability methods for text data DiCE Diverse Counterfactual Explanations Blackbox models: Model formats: Python models using scickit predict convention, Scikit, Tensorflow, Pytorch, Keras Explainers: SHAP, LIME, Global Surrogate, Feature Permutation Glassbox Models: Model types: Linear Models, Decision Trees, Decision Rules, Explainable Boosting Machines AzurML-interpret AzureML SDK wrapper for Interpret and Interpret-community https://github.com/interpretml

Interpretability approaches Glassbox models Blackbox explanations

Fever? Internal bleeding? Stay home Stay home Go to hospital Models designed to be interpretable. Lossless explainability. Glassbox models Decision trees Rule lists Linear models Explainable Boosting Machines

Explain any ML system. Approximate explainability. Blackbox explanations Model Explanation Perturb inputs Analyze Shap Lime Partial dependence Sensitivity analysis

Responsible ML resources Microsoft Responsible AI Resource Center https://aka.ms/RAIresources Azure Machine Learning https://azure.microsoft.com/en-us/services/machine- learning/ https://docs.microsoft.com/en-us/azure/machine- learning/concept-responsible-ml Responsible Innovation Toolkit https://docs.microsoft.com/azure/architecture/guide/resp onsible-innovation FairLearn https://github.com/fairlearn https://aka.ms//FairLearnWhitepaper https://docs.microsoft.com/azure/machine- learning/concept-fairness-ml InterpretML https://github.com/interpretml https://aka.ms//InterpretMLWhitepaper https://docs.microsoft.com/azure/machine-learning/how- to-machine-learning-interpretability

Understand Interpretability Fairness Azure Machine Learning Responsible ML Control Audit trail Datasheets Protect Differential privacy Confidential machine learning

Differential Privacy for Machine Learning and Analytics https://github.com/opendifferentialprivacy Native Runtime C,C++, Python, R Validator Automatically stress test DP algorithms Data Source Connectivity Data Lakes, SQL Server, Postgres, Apache Spark, Apache Presto and CSV files Privacy Budget Control queries by users

WhiteNoise Privacy Module Report Budget Store BUDGET User Private Dataset Submits a query Receives a differentially private report Mechanism adds noise Private data Dataset checks budget and access credentials Checks budget and private compute Credentials to access the data https://github.com/opendifferentialprivacy

Confidential ML Technologies Privacy Barrier Privacy Barrier Confidential from Data Scientist Encrypted using Hardware

Homomorphic Encryption Decrypt(Encrypt(A) + Encrypt(B)) = A + B Decrypt(Encrypt(A) * Encrypt(B)) = A * B Privacy Barrier Homomorphic encryption allows certain computations to be done on encrypted data, without requiring any decryption in the process: Different from classical encryption like AES or RSA:

Microsoft SEAL Open-source homomorphic encryption library Developed actively since 2015 Recently released v3.5 Available at GitHub.com/Microsoft/SEAL Supports Windows, Linux, macOS, Android, FreeBSD Written in C++; includes .NET Standard wrappers for public API From open source community: PyHeal (Python wrappers from Accenture) node-seal (JavaScript wrappers) nGraph HE Transformer (from Intel)

Private Prediction on Encrypted Data Through a trained machine learning model, private prediction enables inferencing on encrypted data without revealing the content of the data to anyone. Microsoft SEAL can be deployed in a variety of applications to protect users personal and private data Privacy Barrier [cryptographic] Medical prediction

Responsible ML Resources  Microsoft Responsible AI Resource Center  https://aka.ms/RAIresources  Azure Machine Learning  https://azure.microsoft.com/en- us/services/machine-learning/  https://docs.microsoft.com/en- us/azure/machine-learning/concept- responsible-ml  OpenDP  http://opendp.io/  https://twitter.com/opendp_io  WhiteNoise  https://github.com/opendifferentialprivacy  https://docs.microsoft.com/azure/machine- learning/concept-differential-privacy  https://docs.microsoft.com/azure/machine- learning/how-to-differential-privacy  https://aka.ms/WhiteNoiseWhitePaper  SEAL  https://github.com/Microsoft/SEAL  https://docs.microsoft.com/azure/machine- learning/how-to-homomorphic-encryption- seal  https://aka.ms/SEALonAML

AI at Scale

Next-gen AI capabilities To accelerate the change we needed, we took advantage of three trends Transfer learning Large pre-trained self supervised networks Culture shift

Microsoft Turing NLG 5 b 7.5 b 10 b 12.5 b 15 b 17.5 b Spring ‘18 Summer ‘18 Autumn ‘18 Winter ‘19 Spring ‘19 Summer ‘19 Autumn ‘19 Winter ‘20 2.5 b ELMo 94m GPT 110m BERT - large 340 m Transformer ELMo 465m GPT-2 1.5b MT-DNN 330m XLNET 340m XLM 665m Grover-Mega 1.5b RoBERTa 355m DistilBERT 66m MegatronLM 8.3b T-NLG 17b Number of parameters

Custom capabilities through reuse Turing NLP

Microsoft Bing Intelligent Answers which summarize the web

Microsoft Word Smart Find Search inside your document as easily as is to search the web

Microsoft SharePoint Inside Look Find that document you’re looking for by browsing automatic document summaries

Microsoft Outlook Suggested Replies Save time by responding with AI assisted suggestions

AI at Scale for everyone

Collaboration with OpenAI Hosted in Azure 285,000 CPU cores, 10,000 GPUs, 400 Gbps for each GPU server Top 5 in Top 500 SCs https://blogs.microsoft.co m/ai/openai-azure- supercomputer/

Learn more: • AI at Scale introduction: aka.ms/AIatScale • AI at Scale Deep Dive: aka.ms/AIS-DeepDive • DeepSpeed library: github.com/microsoft/DeepSpeed • ONNX Runtime: aka.ms/onnxruntime • Try T-NLG in the Companion App: aka.ms/Build-AIS

Rethinking the AI Stack NVidia GPUs Intel FPGAs NVLink Infiniband DeepSpeed allowing for training models 15x bigger, 10x faster on the same infrastructure ONNX Runtime Central AI group coordinating bringing the best of research into products All available on Azure and GitHub for everyone!

© Copyright Microsoft Corporation. All rights reserved.