Circle of Excellence : Préparer son infrastructure pour OpenAI

Transcript

1. 5 Février 2024 Prérequis et choix d’infrastructure pour déployer l’IA générative
2. Introduction Sandra Calabre Azure Sales Lead @Microsoft France Frank Thiry Infrastructure & Application Lead @Avanade Christophe Jeantet Technology Lead @Accenture France
3. Your Organization Where are you today and where do you want to be tomorrow? A I M A T U R I T Y AI READY AI LEADER A I N E E D USING AI BUILDING AI Developing AI Evaluating AI Implementing AI Preparing for AI
4. The Shift to AI requires AI Readiness for 3 reasons BUILDING AI T O USING AI F R O M 1 S T U S E C A S E 3 0 + U S E C A S E S Using Your Data S C A L E Collocated Database(s) Web experience Build Intelligent App(s) as a Service Management & Cost: Compute, Network Security etc. DATA APPS INFRASTRUCTURE
5. Getting AI Ready is a holistic technology conversation beyond Gen AI, LLMs and Machine Learning Data Platform Infrastructure Data Data model of structured and unstructured data Collocating the data with algorithm for congruency, low latency and security Management & Cost Scalable Infrastructure Mgmt. – compute, network and security Managing IT cost effectively and transparently no matter where it sits AI Platform AI, LLMs and ML App Platform Apps Intelligent apps require app modernization Collocating the app and algorithm sees 2x faster throughput per GPU Security Platform From data integrity and data security to securing your AI and Applications estate from sophisticated threat vectors
6. And so We Help You End to End From Infrastructure to AI, across the stack Technology Infrastructure We can help assess your systemic readiness AI Strategy & Solutions We can help accelerate your AI needs & outcomes Azure Migrate & Modernize Assess, trial and modernize your apps and infrastructure estate with partners. Learn More Microsoft Migration Factory Assess, trial and migrate your infrastructure workloads to the cloud Learn more Management Best Practices From cloud management to cost management. Learn More. Azure Innovate Our programmatic approach to help you go from ideation to solution, with Microsoft and our system integrator partners ready to tackle the use cases for your business. Learn More
7. Agenda Vision prospective - les prérequis pour développer at Scale l’IA générative Comment l’IA générative va enrichir, optimiser la gestion des infrastructures et accélérer la modernisation des applications ? Diner au Paris-Seattle 2 3 4 5 Retour d’expérience Gen AI Fundation - Groupe des métiers de l’eau, des déchets et de l’énergie. 1 Cocktail - Networking 20 minutes 30 minutes 20 minutes 30 minutes
8. Vision prospective : les prérequis pour développer at Scale l’IA générative Frank Thiry Infrastructure et Application Lead @Avanade Gregory Abisror AI & GenAI Lead @Accenture 1
9. Prospective Vision : Prerequisite to develop Generative AI at scale GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
10. GenAI found its users and being ready 1
11. By 2024, there's no doubt about it: GenAI is going to change every sector of activity. 1 Data quality & availability New set of skills and expertise Acculturation and Change management Cloud & scalability Editor roadmap monitoring Data Governance & Responsible AI Data quality & availability Acculturation and Change management Editor roadmap monitoring Data Governance & Responsible AI 54% 48% 36% 40% 43% 33% 34% 32% 28% 30% 26% 30% 26% 28% 27% 25% 26% 24% 24% 20% 12% 14% 21% 14% 9% 13% 7% 9% 11% 9% 13% 6% 8% 6% 6% 8% 6% 6% 5% 5% 24% 26% 28% 29% 14% 21% 12% 19% 33% 35% 20% 13% 16% 15% 15% 17% 14% 13% 14% 11% 10% 12% 15% 18% 34% 33% 46% 39% 27% 26% 41% 50% 50% 50% 52% 50% 54% 57% 56% 64% 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Potential for Automation Augmentation (high) Augmentation (low) Non-language tasks a Work time distribution by industry and potential Llearning impact based on their employment levels in the US in 2021 Gen AI is booming and will soon become table-stakes. Every industry is experimenting 43% GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
12. Even IT departments are looking to GenAI to improve the efficiency of their services. Documentation Identify documents that need updating after a task is complete Generate a draft Summarize requirements “Living documentation” Semantic search Testing Redact Cucumber tests using existing or new expressions Draft unit tests, integration tests Improve tests by identifying gaps and edge-cases Generate test data Coding On-demand code generation Recommend optimizations Code completions Locate risk-prone areas Migration between languages and technologies Requirements Transcribe, summarize interviews, extract requirements Analyze and categorize stakeholder inputs App Evolution Identify focus areas for refactoring or risk mitigation Highlight improvement options by analyzing helpdesk, metrics and usage patterns Facilitate dependency and framework upgrade Work-items Generate acceptance criteria from requirements Redact a “What’s new” for a version release Use classification and semantic to vectorize work items Help Desk Automate Q&A for case qualification and investigation Recommend responses to support agents Route requests to appropriate teams Project Mgt Redact status reports on ongoing developments Identify upskilling opportunities GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
13. But Generative AI also exacerbates the standard foundational needs of AI … to deliver its full value potential 3 Generative artificial intelligence Business process reinvention Data quality & availability New set of skills and expertise Acculturation and Change management Security & dependency to external providers Cloud & scalability Editor roadmap monitoring Data Governance & Responsible AI GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
14. From Prototype to Production 2
15. Scalable foundations The most ambitious use-cases, up to Enterprise-scale solutions Nullifying tech waste Azure Open AI Security (Exposure and IAM) Cost Management Usage Tracking Private ChatGPT / Chat use-cases API For developers Infra-as-Code Workshops Gen AI Foundations Upscale to... Experimentation sandbox Chat-based use-cases Applications improved with GenAI Work on POC and prepare MVP Use-case roadmap – LLMs benchmarking Gen AI for IT use-cases 1 Prompt flow engineering Context-sensitive prompts Gen AI-native solutions Data processing Orchestration Classification Corporate data integration Vectorization / Semantic expansion Document repository workflows Custom data embeddings Enterprise-scale Pervasive AI 3 Model training & fine-tuning Model maintenance and evolution Specialized content generation Training datasets ML Ops – LLM Ops 2 4 Growth driven by complexity ramp-up GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
16. GenAI – Priority to learning and capitalizing From Experimentation to Production Relevant Indicators for Evaluation MS Technology Stack • Azure OpenAI • Azure Cognitive Search • Azure Document Intelligence • Python/Spark • JupyterHub, VsCode • Foundation : Gaps between sources and results • Relevancy : Gaps between results and the questions • Recovery : Gaps between sources and questions • Performance (UX) Experimentation L1 Experimentation L2 Préprod & prod Learning Capitalizing Demand Management Sample Test Test requiering specific Features Use Cases Azure OpenAI Studio Azure OpenAI Front Custo Azure OpenAI Development Azure OpenAI Development Specific case Referential Feeding GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
17. Technical blueprint  Solution design & extensibility pattern Key Vault Tags Cost Mgt Log Analytics App Insights Monitoring Managed Identities AI Services Open AI Machine Learning Other Azure hosted LLMs Azure AI Resources Monitoring | Observability Cosmos DB for PostgreSQL Cognitive services Supporting services RAG Services Processing Storage Application Layer Web applications, REST APIs API Management App Gateway Web App Firewall Document base ... ... Data Web interface Azure Bot Exposure Layer Microsoft Teams Active Directory Entra ID Monitoring | Observability • Common services securing information and tracing activity Azure AI Resources • Machine Learning, Artificial Intelligence and Cognitive Services • Other LLMs hosted on dedicated infrastructure RAG Services • Document and data records chunking, vectorisation and storage in a vector database (PostgresSQL + pgvector, Chroma, VectorDB...) Document base • Internal document and data repositories including files (Word, Pdf, Excel, Calc, Slides, Pages...) and data from SQL and DocumentDb sources Application Layer • Application hosting querying vectors and feeding identified chunks to AI Services Exposure Layer • Securing access across different user interfaces • Web applications • Teams integration... GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
18. Architecture Principles Principles GenAI models are components, not full applications ML Ops techniques are a must to scale GenAI (in production) GenAI is only as effective as your architecture and data foundations Relevant Generation heavily depends on prompts, knowledge bases and fine-tuning Generative AI models must be filtered when used in platforms Choose the right AI model for the right business task GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
19. Detailed Architecture Principles Principles Large Language Models, like GPT-4, are versatile in Natural Language Processing, excelling in tasks from sentiment analysis to text generation. However, their use comes with performance, pricing, and environmental considerations. Model training emits a substantial amount of carbon emissions, emphasizing the importance of selecting the right model for the specific use cases. Choose the right AI model for the right business task AI generative models are crucial components in real-time transactional applications like E-commerce. They must provide high availability, scalability, and resilience. For instance, in AI-enabled E-commerce, these models detect customer intent, leading to product recommendations and order management. GenAI models are components, not full applications Generative AI systems require continuous monitoring through MLOps KPI and techniques to detect and correct training biases in near-real-time, ensuring system fairness and performance. ML Ops techniques are a must to scale GenAI (in production) GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
20. Detailed Architecture Principles Principles To prevent biases and inaccuracies in downstream components of the architecture, Generative AI responses should be carefully filtered and processed by responsible AI elements. For instance, in AI-driven E-commerce, the recommendation engine should align with customer intent detected by the LLM, ensuring relevant and aligned vocabulary usage. Generative AI models must be filtered when used in platforms Out-of-the-box Generative AI models may not align with specific business objectives. Instead of full retraining models, use a query system with advanced prompt engineering, coupled with a well-maintained knowledge base. Fine-tuning techniques like LoRA can also be beneficial, requiring marketing and content management teams to acquire new skills to ensure the gen AI model knowledge. Relevant Generation heavily depends on prompts, knowledge bases and fine- tuning Generative models mimic knowledge base characteristics they interact with, making the quality of their outputs reliant on the knowledge base's quality in terms of history, data cleanliness, and automation. Rich training base allows smaller models to match the performance of larger, resource-intensive models trained on lower-quality data sources. For instance, a Luxury Maison with a robust Data Architecture can draw from decades of data to generate relevant results. GenAI is only as effective as your architecture and data foundations GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
21. GenAI Architecture blueprint Corporate Landing Zone ( Network, Container Hosting, Security, Monitoring…) GenAI Landing Zone (Data Projects, Data Fabrick, , Vectors DB, Orchestration, LLM Monitoring ...) GenAI Software ( LLM Calls, Retrieval LLM Chain Prompt LLM Chain, PII Management, Control rules & Monitoring … ) Use-Cases ( Quality Issues Non- Conformity Similarity … ) Corporate Technical and Infrastructure Foundation ( Cloud & On-prem ) Infrastructure Foundation Software Foundation Templates Use-Cases GenAI Template ( Chatbot, Summarization, Search … ) Conf to Products Industrialized GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
22. Architecture Templates Considerations RAG & MultiModal RAG – no document quote RAG + structured data access RAG + structured data access + application API call RAG + Image analysis (ex IT support assistant) RAG + document quote, enabling voice transcript (in/out). RAG + structured data access, enabling voice transcript (in/out) Augmented agent: chatbot with RAG + structured data enabling 2 ways voice transcript Code Assistance​ Level 1: code generation using coding LLM Level 2: code generation/document/reingeneering enabling usage of existing code / templates Data Analysis ​ Data analysis on structured data Unstructured data understanding and analysis​ Document analysis solution – solution analyzing text for specific purpose: intent detection, summarization, …. Content generation​ Text – solution that enables prompt-based content generation Text – solution that enables content generation based on existing documents to support marketing, training, policies or contract generation Image - solution that enables prompt-based content generation Image - solution that enables content generation based on existing documents to support marketing, training documents/content generation Architecture Generation, 3D Design, … GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
23. GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion Important QUESTIONS for implementing gen ai FUNCTIONAL 1) Which is the role of AI into my system/application: which application tasks/use cases could be addressed by Generative AI? 2) How will users interact with Generative AI? 3) How should I ethically manage Generative AI outputs? 1 APPLICATION (Architecture Building Blocks) 1) How will the gen AI component interact with other system components: which protocol (vectorization/embedding, internal protocol) ? 2) Do we have to train it? With which data? Is it a one-shot training or a recurrent? How should we take care and avoid training biases ? 3) Which knowledge base should it rely on to provide/create answers? • Is training base enough or should it be updated continuously ? • Which link between training base and knowledge base? 4) Does it need a memory to remember past interactions ? 5) How can we manage the prompt content to get relevant answers ? 2 TECHNOLOGY (Solution Building Blocks) 1) Which technological components should we use (Azure, OpenAI, custom, Open Source) ? 2) How can we guarantee the scaling of the overall Gen AI based platform ? 3) How should we continuously monitor Generative AI answer quality and act accordingly? 3
24. Model use best practices 7 Dynamic routing to optimal model 8 Benchmarking Engine 9 Integrated Controls 10 Model Registry 11 New Model Integration 12 Model Operations Review pre-approved models for an application team Rules engine that routes to the appropriate model Review a model’s benchmarks and usage guidance for making initial decisions Recommend prompts tailored for use with specific models A collection of well defined and measurable KPIs for routing decisions Rules repository for capturing routing decisions Compare model performance using standard & tailored measures Manage model endpoints and configuration Capacity and availability management of model endpoints Seamless plug-n-play model enablement Collect usage and performance metrics for models Governance controls for model endpoints 1 Catalog of approved models and benchmarks 2 Model usage guidance 3 Prompt generation assistance for models 4 Prompt KPI comparison between multiple models 5 Catalog of available KPI measures 6 Codification of routing decision Exploration Governance Decision Making Optimization Governance GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
25. Security as a cornerstone 3
26. Security as a cornerstone Tempering with raw data Contaminating input databases (e.g. Training and validation) Potential for altering PII Missing critical data Rapid pace of AI applications Overfitting data Tempering with underlying algorithms Attacks on the model Attacks on classifier: • Targeted • None targeted Threats becoming increasingly difficult to recognize People Altering output data Feedback loops External interaction between connected processes and data stores Software Risks Communication Risks Data Risks System Risks Human Factor Risks GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
27. Comprehensive Security Standards Are Critical 2 DATA PROTECTION SANDBOX SECURITY APPLICATION SECURITY COMPLIANCE AND REGULATORY REQUIREMENTS CYBERSECURITY, RISK AND INCIDENT MANAGEMENT/RESPO NSE Measures and strategies used to safeguard sensitive information from unauthorized access, theft, or misuse. Measures taken to secure the physical and digital sandbox components Ensure the security, confidentiality, and integrity of machine learning models and their associated applications throughout their lifecycle Set of standards, guidelines, and legal obligations that govern the use, handling, and processing of data and machine learning models. Identifying, evaluating, and mitigating risks including cyber threats to an organization's assets, including data, systems, and processes. KEY STANDARDS • Encryption and key management • Data retention policies and procedures • Data backup and recovery • Network security controls • Firewall and intrusion detection/prevention systems • Security monitoring and incident response • Code reviews and testing • Vulnerability scanning and penetration testing • Access controls and authentication mechanisms • HIPAA, GDPR, CCPA, etc. compliance • Industry-specific security standards (e.g., ISO 27001, NIST, PCI DSS) • Risk assessments and mitigation plans • Security incident management and response procedures GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
28. Gen AI Security Concerns and Ways to mitigate Security Concerns Mitigation Solutions Productionizing AI MLOps, DevOps, DevSecOps Sandbox/Segmentation Production ready kick-starter templates Confidential Information Leak – Privacy Prevention AI/GenAI Assessment Toolkit Data Privacy and cyber frameworks, policies and auditing procedures Immutable assets and Filters Adversarial ML Attack Data Sanitisation; Model Segmentation Counterfit - Model vulnerabilities Pentesting Data Governance Prompt Filtering; Local ML Models: Data governance Assessment Legal Concerns Internal and external legal review IP confidentiality ownership Ethics Concerns Responsible AI Assessment, GenAI Assessment Consider Amplification of Bias and how to mitigate. Prompt Injection Attacks MLOps, DevSecOps and all Engineered Prompts review and remediate Monitoring SIEM integration (e.g. Sentinel) GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
29. Enterprise Security Frameworks Need To Be Enhanced LLM Threat Types Adversarial Attacks BIAS Model Leakage/stealing Model Poisoning Responsible Use Malicious Use Enterprise Extended Security Framework (LLM) Training Data Threat Modeling Business Context Language Translation Model Training Monitor Model Verification Data Sanitization Enterprise Security Framework Identity and Access Control Data Security & Privacy Audit Deployment Standards Infrastructure Network Info security polices and standards A comprehensive LLM security framework, integrated with the broader enterprise security framework can effectively address most known LLM threat types GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
30. Guiding Principles for Responsible AI 4
31. Responsible AI is not just a regulatory requirement it is a commitment to using AI for the greater good 1 Business process reinvention Data quality & availability New set of skills and expertise Acculturation and Change management Cloud & scalability Data Governance & Responsible AI GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion 1 5 3 7 Risk Mitigation: Ethical governance helps mitigate legal, financial, and reputational risks, preventing compliance issues and financial losses tied to unethical behavior. Reputation: Ethical practices build and maintain a positive reputation, fostering trust among customers, investors, and the public. Customer Satisfaction: Ethical business practices directly contribute to heightened customer satisfaction, fostering brand loyalty. Talent Attraction: Ethical organizations attract top talent, as prospective employees seek workplaces aligned with their values. Social Responsibility: Ethical governance reflects an organization's commitment to social responsibility, meeting societal expectations. Trust Building: Ethical governance is vital for building trust with diverse stakeholders, enhancing collaborative efforts. Sustainability: Ethical governance is crucial for long-term sustainability, enabling organizations to adapt and endure. Legal Compliance: Ethical governance regards legal requirements as a baseline
32. Responsible AI encompasses more than just fairness and bias Transparency • Interpretability & Explainability • Understanding • Traceability Soundness • Context & Comprehension • Data Quality • Model Performance & Validation • Inconsistent outputs • Language toxicity Privacy • Data Ethics • Human Rights • PII Privacy & Regulation • Disclosure of confidential information Sustainability • Environmental Sustainability • Human-centric design • Socio-economic wellbeing Fairness • Anticipating bias • Encouraging Diversity Accountability • Governance • Human Oversight • Auditability Robustness • System Security & Safety • System Resilience • Response Plan Liability and Compliance • IP Ownership Concerns • Contractual and Product Liability • Consumer Protection • Evolving Regulatory Compliance Landscape Key: Purple items expand on existing RAI Dimensions 8 Dimensions of Responsible Artificial Intelligence GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
33. 33 Example of RAI end-to-end infrastructure and services components 33 Experience Plane Risk Assessment User inputs to assessment questionnaire based on compliance policies Dashboards View outputs of explanations, biases, data quality, and model robustness detection, and approve any necessary remediations Testing Services One-time/on-demand detection and mitigation of RAI factors, like bias, explainability, robustness and PII Continuous RAI Monitoring Services Continuous monitoring and mitigation of RAI services through integration with Data Ops and MLOps Integration Plane API Gateway Handle API communication between client and API microservices Orchestration Workflows, notifications, batch/NRT job orchestration Software Development Kits Containerized REST APIs/SDKs for invoking questionnaires, RAI detection and mitigation, explainability services Connectors Third-party tool connectors, APIs for data sources, model pickle files, and prompt databases Services Plane Risk Assessment Services Risk assessment based on responses to the questionnaire RAI Mitigation Services Mitigate RAI factors, including data/prompt/model bias, data quality, robustness Data Privacy Services Policy engine, PII detection, annotation and tagging services. Data protection services RAI Monitoring Services Data, model and prompt bias detection, data quality, robustness detection Explainable AI Services Model explainability, including deep learning-based models Services Plane Results Store Storage of outputs from the services plane (risk assessment, RAI detection, RAI factor mitigation and explainable AI services) AI Use Case Metadata Store Pipeline, workflow hierarchy information, approval status, risk assessment questionnaire etc. Metrics Store RAI metrics, model metrics (accuracy, Bleu, Rouge etc.) GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
34. Building Pillars of Governance 5
35. (Gen)AI CoE Executive Sponsor Key elements are essential in establishing a (Gen)AI CoE 35 Steering Committee 1 2 3 C-suite executive sponsorship to ensure accountability and E2E visibility Steering Committee members set strategic direction and risk tolerance of the Generative AI organization. Provide guidance on organizational policies, funding, and escalations. Should include the leadership and senior executives from HR, Legal, Finance, Operations, IT, Data and AI and the leads of the business lines (Gen)AI CoE co-led by both technology and business to ensure fit-for-purpose model outcomes Lab Activation pods assembled with multidisciplinary roles to execute AI initiatives 4 1 2 3 4 Strategy & Governance Methodology Risk Management & Ethics Demand Management AI Platform & Tech Enablement Resource Allocation AI & Model Ops Talent & Learning Data Factory & Ops Innovation Management & Tech Watch Delivery Squads Communication & Reporting Communication, reporting & value measurement 5 5 GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
36. Conclusion 6
37. Accenture & Avanade teams can at the same time deliver solid GenAI foundations & use cases 7 Business process reinvention Data quality & availability New set of skills and expertise Acculturation and Change management Cloud & scalability Data Governance & Responsible AI New AI use case with high business value GenAI Use Case Strategy Data governance (role, responsibility) with the implementation of a responsible AI approach Data Governance & Change Building data platforms (move to cloud) adapted to AI and building high-performance Gen AI architecture (FinOps, Cloud providers) Platform & FinOps • Data office setup and operating model • Data management & quality • Responsible AI framework • Training & Acculturation (tbd) • Data & AI strategy • Sector and functional expertise • Gen AI products & features • Benchmark use case / industry Accenture Assets Accenture Assets • Gen AI Architecture • FinOps Gen AI • Data & AI Platform • Partnerships Accenture Assets GenAI found its users and being ready 1 3 5 2 4 6 From Prototype to Production Security as a cornerstone Building Pillars of Governance Guiding Principles for Responsible AI Conclusion
38. Retour d’experience Gen AI Foundation Michel Hubert Solution Architect @Avanade 2
39. Private and secure Open AI solution for end-users and application developers CEO • Prohibits the use of public GenAI Services BUs • IT urged by business units to roll out a GenAI solution to leverage GenAI Capabilities • Avoid Shadow AI CIO • Support to define and build architectural foundations fo a v0 GenAI solution that aims to direct users calls to an internal and secure GenAI service leveraging Azure OpenAI
40. Starting GenAI journey with a secure solution combining the power of OpenAI and Azure Centralized service administration and monitoring Two features accessed from a single Azure subscription OpenAI A conversational app leveraging OpenAI pre-trained models capable of creating content (language and code) OpenAI API An API hosted on Azure giving access to OpenAI pre-trained models (ChatGPT in PoA) Feature 1 Feature 2 Experience instant answers on almost everything Boost inspiration and productivity Acquire new knowledge Optimize, correct and translate code to natural language Augment existing applications Build new applications Retrieve real-time information Fine-tune models (coming in next iterations) GenAI App users App developers GenAI Service admins User persona
41. Security with no compromise  Shared responsibility principle  Scope of security  Access and Identity  Network (firewall, IDS, …)  Endpoint management (API Management)  Data Security  Sensibilize and train employees :  Understand concepts  how to use basic & advanced “prompt”
42. PoA Designer choices at a glance: a SOLID SaaS solution fully implemented on Azure GenAI Service Admin Log Analytics Monitor Azure Subscription Context Private Context Azure Application Gateway + WAF Prompt Completion GenAI App User Azure APIM App Developer Consumer Access Configuration Global Services Configuration Monitoring Key Vault Azure WebApp* Consume API Consume API Consume API Prompt Completion Monitoring Services Azure OpenAI Models GPT-3.5-turbo Read-only Internet snapshot (09/2021) Identity provider: • GenAI App Users and Service Admins are authenticated using SSO • Developer Apps are authenticated using API Keys 1 Azure Application Gateway + WAF ensures SSL offloading and DDoS protection 2 Azure APIM is used to implement fine-grain access, enforce monitoring and store product configuration and associated keys 3 Azure WebApp hosts and exposes the OpenAI sample application within the private context 4 Key Vault is used to store secrets and certificates (e.g. connection strings, API keys) 5 Azure Monitor, Log Analytics and Storage accounts services are used to collect and monitor KPIs (retention up to 1 year) 6 1 2 3 4 5 6 Solution is ISO 27001:2013 compliant ** recurring or severe abuse may result in service suspension by Microsoft
43. Next Steps…
44. Comment l’IA générative va enrichir, optimiser la gestion des infrastructures et accélérer la modernisation des applications ? Eric de Smedt Technical Specialist Azure Infra @Microsoft France 3
45. Microsoft Copilot for Azure  February 2024 +
46. ©Microsoft Corporation Azure Streamline cloud operations and management with generative AI
47. ©Microsoft Corporation Azure Helping for design resilient solution Microsoft Copilot for Azure (preview) can help you deploy virtual machines in Azure that are efficient and effective. You can get suggestions for different options to save costs and choose the right type and size for your VMs.
48. ©Microsoft Corporation Azure Build Infrastructure and deploy workloads Microsoft Copilot for Azure (preview) can help you quickly build custom infrastructure for your workloads and provide templates and scripts to help you deploy
49. ©Microsoft Corporation Azure Secure and protect your workloads Microsoft Copilot for Azure (preview) can provide contextual and dynamic responses to harden the security posture and enhance data resiliency of storage accounts.
50. ©Microsoft Corporation Azure Analyze costs Microsoft Copilot for Azure (preview) can help you analyze, estimate and optimize cloud costs. Ask questions using natural language to get information and recommendations based on Microsoft Cost Management.
51. ©Microsoft Corporation Azure • Help with documentation or testing • Support in learning a new language • Accelerating the resumption of a project • Code modernization, application debt management • Assistance with code reviews • Peer programming
52. Call-to-action Microsoft AI Tour Scannez ici pour vous inscrire au Jour 1 13 mars Pour les profils techniques 12 mars Pour les décisionnaires 📅 12-13 mars 2024 📍Carrousel du Louvre