qaware.de Turbocharging AI Innovation How AI Platforms Enable The Bulletproof Deployment of GenAI Use Cases Mario-Leander Reimer © 2024 QAware

2 Mario-Leander Reimer Managing Director | CTO @LeanderReimer #cloudnativenerd #qaware #gernperDude

The Next Frontier The Vast Universe of AI Use Cases

Welche KI Use Cases sind in euren Unternehmen geplant oder auch schon umgesetzt? ⓘ Click Present with Slido or install our Chrome extension to activate this poll while presenting.

Use Case: Customer Support Call Chat Support Resources Knowledge Database Support Flow Customer Data Images and Icons were generated with the assistance of AI

Use Case: Customer Support Support Resources Knowledge Database Support Flow Customer Data Images and Icons were generated with the assistance of AI RAG Automation Intent Recognition Text2Speech Speech2Text Anomaly Detection Similarity Matching AI Assistant Call Chat

Use Case: Content Generation Simple and easy generation of ■ E-Mail responses ■ Product videos ■ Presentations ■ User manuals ■ Document summaries ■ Tender documents ■ Source code in the desired style. Images and Icons were generated with the assistance of AI

Use Case: Anomaly Detection, Predictive Maintenance, ... Image Source: https://azure.microsoft.com/en-us/products/ai-services/ai-anomaly-detector

The most common use cases for Gen AI Tools are marketing, sales, product development, and service operations. Source: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year

Wie funktioniert eigentlich ein Chatbot? Was steckt hinter RAG, Embeddings, .. ?

Wie funktioniert Retrieval Augmented Generation (RAG)? Index, z.B. Vector DB Indexing (Chunking & Embedding) Dokumente Ingestion Phase Query Encoding Retrieval Phase Context Prompt LLM mit Weltwissen Response

Vom Input zum Embedding. Das ermöglicht die performante semantische Suche mittels Vector Datenbanken. Embedding Model Images were generated with the assistance of AI { 23.567, 45.899, 76.345, …}

Wenn das nicht reicht, hilft Transfer Learning. Pre-trained Model Gut genug? Re-Training mit spezialisierten Daten Spezialisiertes Model Task ja nein Images were generated with the assistance of AI

Why do we need an AI platform?

But we are already doing this! … Really? MLOps only covers certain parts of the tasks around GenAI. Source: https://neptune.ai/blog/mlops https://dl.acm.org/doi/10.5555/2969442.2969519

Each involved stakeholder has a different expertise and therefore a different focus. These must be consolidated. Domain Expert Software Engineers and Architects This image was generated with the assistance of AI Data Scientists, AI Experts Platform Engineers

Models and talent pose the biggest challenges, alongside strategy as a frequent hindrance. Source: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year

Your data and business domain are the driving force. Start here. CRISP-DM helps to start a structured AI approach. Business Understanding Data Understanding Data Preparation Modelling Evaluation Deployment Source: Shearer, C. (2000). The CRISP-DM model: The new blueprint for data mining. Journal of Data Warehousing, 5(4), 13-22

Chatbots and AI Assistants: Options and Development Stages ChatGPT or comparable with world knowhow ChatGPT with organisational context knowledge Specialized AI Assistent ■ Retrieval Augment Generation ■ Transfer Learning ■ Custom trained modell ■ Process Automation Complexity Benefit ■ Easy to use and cost efficient ■ Needs guidelines on data protection & compliance

“Too much cognitive load will become a bottleneck for fast flow and high productivity for many teams.” ■ Intrinsic Cognitive Load Relates to fundamental aspects and knowledge in the problem space (e.g. languages, APIs, frameworks) ■ Extraneous Cognitive Load Relates to the environment (e.g. console command, deployment, configuration) ■ Germane Cognitive Load Relates to specific aspects of the business domain (aka. „value added“ thinking)

Platform engineering is the discipline of designing and building toolchains and workflows that enable self-service capabilities for software engineering organizations in the cloud-native era. Platform engineers provide an integrated product most often referred to as an “Internal Developer Platform” covering the operational necessities of the entire lifecycle of an application. https://platformengineering.org/blog/what-is-platform-engineering

A platform consists of different conceptual components. Depending on the stakeholders and their use cases. Developer Control Plane Integration and Delivery Plane Monitoring and Logging Plane Security Plane IDE Service Catalog / API Catalog Developer Portal Application Source Code Infrastructure & Platform Source Code Observability Secrets & Identity Manager CI Pipeline Registry CD Pipeline Resource Plane Compute Data Integration Networking Platform Orchestrator Certificates & Encryption GitOps https://humanitec.com/reference-architectures

Our proposal for an AI Platform Architecture

AI platform engineering is the discipline of designing and building toolchains and workflows to provide self-service capabilities for data and AI driven organizations. Business experts, data engineers as well as software engineers work together in an integrated platform from now on referred to as an “Enterprise AI Platform” covering the operational necessities of the entire lifecycle of AI use cases. © 2024, M.-Leander Reimer

Integration & Delivery Plane Service Plane Quality Plane Data Plane Platform Plane Observability Operability Resource Plane User Serving Plane Access Plane / APIs Orchestration Plane Data Modelling Plane Model Plane Compliance Plane Compute Data Integration Security Delivery FinOps

Quality Plane Integration & Delivery Plane Service Plane Access Plane/APIs User Serving Plane Technical and Business Metrics like Accuracy, Harmfulness, … Test Automation for LLMs „Convenience UIs“, Self Service, RAG per Drag and Drop, … (a) LLM, Embedding, (b) RAG, Chatbot, … (c) Data Access, … Orchestration Plane Data Modelling Pl. Playground Prompt Engineering Konfiguration Runtime, Instantiation, Orchestration, Scaling, Configuration Data Plane Ingestion Pipelines Data Versioning Embeddings & Vectorization Model Plane MLOps: Model Registry Model Management Experiment Tracking Model Serving Compliance Plane Tonality, Bias Security, Data Protection Platform Plane Observability: Monito- ring, Logging, Tracing Security: Secrets, IAM Encryption, Certs, … Scale, Backups, Recovery, … Delivery: CI/DC, Registry Pipelines, Orchestrator, … FinOps Resource Plane Compute: CPU and GPU Data: Vector DBasS, other Storage, … Integration: Self-hosted LLMs Public LLMs Managed AI Services

From concept to realisation: Possible variants

Many roads lead to Rome. Depending on your context, one or the other makes sense. Buy an AI platform solution + Convenience - Possibly not 100% suitable or extensible - Vendor Lock In Combination of cloud provider building blocks + Easily available - Vendor Lock In - Data Protection Considerations Custom platform with open source components + flexible and fully customizable - Time and Money for Setup and Maintenance

Azure AI Studio (Preview) Azure AI Content Safety Quality Plane Integration & Delivery Plane Service Plane Azure API Management Access Plane Azure AI Studio (Preview) User Serving Plane Azure AI Studio (Preview) Semantic Kernel Orchestration Plane Azure AI Document Intelligence Data Modelling Pl. Azure AI Search with Indexers, Indices incl. Vector DBs. OneLake, Fabric Data Plane Azure OpenAI Azure Machine Learning Model Plane Azure AI Content Safety Compliance Plane Platform Plane Observability Security Scale, Backups, Recovery, … Delivery FinOps Resource Plane Compute Data Azure OpenAI Azure AI Language Speech Service Azure AI Translator Integration Overview on Azure AI Services: https://learn.microsoft.com/en-us/azure/ai-services/what-are-ai-services

mlflow, Evidently AI, RAGAS (for RAG), DeepEval (for LLM) Quality Plane Integration & Delivery Plane Service Plane API Gateways Access Plane Build your own User Serving Plane Kubeflow Orchestration Plane Jupyter Kubeflow Data Modelling Pl. Weaviate, neo4J, … Custom Pipelines Data Plane mlflow (Registry) BentoML (Serving) Kubeflow (Serving) Model Plane Build your own Compliance Plane Platform Plane Observability Security Scale, Backups, Recovery, … Delivery FinOps Resource Plane Compute Data LLMs: Llama, Mistral, … mlflow BentoML Integration

No content

mlflow, Evidently AI, RAGAS (for RAG), DeepEval (for LLM) Quality Plane Integration & Delivery Plane Service Plane API Gateways Access Plane Build your own User Serving Plane Kubeflow Orchestration Plane Jupyter Kubeflow Data Modelling Pl. Weaviate, neo4J, … Custom Pipelines Data Plane mlflow (Registry) BentoML (Serving) Kubeflow (Serving) Model Plane Build your own Compliance Plane Platform Plane Observability Security Scale, Backups, Recovery, … Delivery FinOps Resource Plane Compute Data LLMs: Llama, Mistral, … mlflow BentoML Integration Build at own risk or better ask an AI platform architect from QAware!

Welche Plattform Variante würdet ihr wählen? ⓘ Click Present with Slido or install our Chrome extension to activate this poll while presenting.

Which solution is the right one for me?

We suggest: Start lean and agile! Only with components required instead of "one size fits all"! Use Case Identification Business Understanding Skill, Resource & Requirements Analysis Building Block Mapping & Prioritization Implementation Evaluation Commoditization

QAware GmbH | Aschauer Straße 30 | 81549 München | GF: Dr. Josef Adersberger, Michael Stehnken, Michael Rohleder, Mario-Leander Reimer Niederlassungen in München, Mainz, Rosenheim, Darmstadt | +49 89 232315-0 | [email protected] Thank you!

We take responsibility and risks: From prototypes to large programs. We deliver. Guaranteed. 1. Our cross-functional teams of consultants, developers and managers see themselves as enablers. 2. We transform your organisation directly through project collaboration. With three guarantees: 3. Guarantee of success: We take responsibility and share your risks, for example through fixed prices. 4. Quality guarantee: You receive sustainable, reliable quality software – documented via KPIs and contractually fixed. 5. Satisfaction guarantee: We tie part of our remuneration to your satisfaction. 200 Engineers Munich Mainz Darmstadt Rosenheim Successful in the most demanding projects for 18 years Cloud Native Transformation & Host replacement: Tour guide into the future Data Value & AI: Open up data, network it & make it valuable From Allianz to Hellmann to Ericsson - we have been instrumental in their transformation. Pure impact: from prediction to AI assistants with valuable data and logic to the AI governance of tomorrow. When your boldest business ideas push the boundaries of IT, we push them together. 35 m € revenue Expertise for you Top provider: NPS 100 Top employer: 97% say "QAware is a very good workplace" Business Booster: Enable & accelerate business-critical visions Guaranteed success