Von der Idee zur Wirkung: Architektur und Aufbau einer Cloud-nativen AI-Plattform #CLC25

Unternehmen erkennen zunehmend die strategische Bedeutung von Künstlicher Intelligenz (KI), doch die Erstellung und Bereitstellung von KI-Lösungen in großem Maßstab ist mit erheblichen Herausforderungen verbunden.

Eine KI-Plattform vereinheitlicht und vereinfacht den gesamten KI-Lebenszyklus. Sie verfügt über einen Cloud-basierten, generischen Kern, der sich um übergreifende Aspekte wie Compliance, MLOps und Monitoring kümmert. Modulare Adapter sorgen für eine nahtlose Integration in die bestehende IT-Infrastruktur.

Dieser Vortrag bietet eine praktische Schritt-für-Schritt-Anleitung für das Design und den Aufbau einer solchen Plattform, sowohl lokal als auch in der Public Cloud. Wir zeigen, wie Kubernetes, Open-Source-Technologien und GitOps-Prinzipien dabei helfen, eine hochautomatisierte und skalierbare Umgebung zu schaffen.

Transcript

1. qaware.de From idea to impact. Architecture and Design of a K8s-native AI platform Mario-Leander Reimer mario-leander.reimer@qaware.de @LeanderReimer @qaware #CloudNativeNerd #gerneperdude
2. 2 Mario-Leander Reimer Managing Director | CTO @LeanderReimer #cloudnativenerd #qaware #gernperDude
3. qaware.de A wave is coming!
4. qaware.de Agentic AI Software engineering agents Domain specific agentic workloads
5. Productivity ... as a strategic imperative. A lack of productivity can be recognised by 1. Non-marketable prices 2. Large backlog of unprocessed orders 3. Low conversion rate 4. Low customer and employee satisfaction Shortage of skilled labour due to demographic change and changing work motivators Competition Marketable prices even when new challengers enter the market Automation Replace or extend human labour with automation Assistance Make human labour more productive Optimisation Recognising potential for improvements in existing processes and workflows
6. Better Assistance and Process Automation with Agentic AI
7. Why do we need an AI platform?
8. "According to Gartner, 80% of AI PoCs fail on their way into productive use." https://www.qaware.de/ki-vom-proof-of-concept-poc-zur-entwicklung/
9. The 80% Fallacy of AI projects. 9 QAware Juan Pablo Bottaro, LinkedIn Engineering Blog
10. Key challenges: technology, models and tools, scaling. Source: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year ▪ Different challenges are seen depending on the maturity of the group ▪ AI newcomers often underestimate the complexity of technologies, models and tools ▪ Production and scaling challenges often hinder production readiness ▪ High cognitive load and lack of expertise are also drivers for failing projects 10
11. "Too much cognitive load will become a bottleneck for fast flow and high productivity for many DevOps teams." Team Topologies: Organizing Business and Technology Teams for Fast Flow
12. "Too much cognitive load will become a bottleneck for fast flow and high productivity for many Agentic AI teams." Team Topologies: Organizing Business and Technology Teams for Fast Flow
13. Big Ball of Agentic Workflow Mud Problem: Complex agentic logic and backend integrations are tightly coupled with workflows, resulting in … 1. poor maintainability and resilience, 2. weak governance & compliance risks due to missing controls, 3. and a high degree of vendor lock-in with fast-moving tech.
14. Agentic AI Lock-in Greediness
15. Our proposal for an AI Platform Reference Architecture
16. qaware.de Agentic AI Domain specific agentic workloads
17. qaware.de ... we have the perfect surfboard! The logical continuation: a. From applications to microservices to AI agents b. From on-prem to cloud platforms to AI platforms
18. Micro-Agent GenAI Usage Prompts, Flow control Tools (MCP) Antwort enthält Aufrufe an OpenAI API ❏ Clear responsibility ❏ Vertical in terms of expertise ❏ manageably large ❏ potentially reusable Micro-Agent A2A AI agents will be implemented according to the microservice architecture paradigm. … … … Tool Server Business Logic LLM, LAM, SLM, domain-specific foundation models ? SSE HTTP
19. An example reference architecture for an IDP. Integration and Delivery Plane Monitoring and Logging Plane Security Plane Observability Secrets & Identity Manager CI Pipeline Registry CD Pipeline Resource Plane Compute Data Integration Networking Platform Orchestrator Certificates & Encryption https://humanitec.com/reference-architectures Developer Control Plane IDE Service Catalog / API Catalog Developer Portal Application Source Code Infrastructure & Platform Source Code GitOps
20. Use your IDP as the foundation for an AI platform. Integration and Delivery Plane Monitoring and Logging Plane Security Plane Observability Secrets & Identity Manager CI Pipeline Registry CD Pipeline Resource Plane Compute Data Integration Networking Platform Orchestrator Certificates & Encryption https://humanitec.com/reference-architectures
21. Platform Plane Observability Operability Resource Plane Compute Data Integration Security Delivery FinOps Integration & Delivery Plane Quality Plane Data Plane Model Plane Compliance Plane Service Plane User Serving Plane Access Plane / APIs Orchestration Plane Data Modelling Plane
23. Compliance Plane Integration & Delivery Plane Service Plane Platform Plane Operability Resource Plane Compute Data: Local SSD Integration Security Delivery FinOps Quality Plane Data Plane Model Plane User Serving Plane Access Plane Data Modelling Pl.
24. lreimer/k8s-native-ai-platform lreimer/k3s-ai-platform
25. https://www.agentic-layer.ai/ https://www.agentic-layer.ai
26. Bring Your Own Model & Cloud Agentic Layer Agentic & Assistive Workforce Agent Runtime (Multi-Framework) AI Gateway (Multi-Model) Bring Your Own Frontends Spacers Agent Testbench Agent Observability Agent Catalog Controls Agent Gateway (Multi-Frontend) OnPrem 1. Convergent: Develop and operate agents and assistants on existing Kubernetes-based cloud platforms. 2. Integrable: Use the same agents and assistants across different workflow systems and conversational UIs. 3. Smart Compliance: Implement compliance requirements intelligently through ready-made building blocks for testing, monitoring, and guardrailing — instead of manual work. 4. Sovereignty: Open source, transparent, and adaptable. Avoid unnecessary coupling to volatile technologies through clean abstraction layers. Connectors
27. The technical backbone for smart workloads.
28. Agentic Layer - High Level Architecture
29. agentic-layer/
30. Bring Your Own Model & Cloud Agentic Layer Workflow Automation The KAN stack: Kubernetes, Agentic Layer, n8n Enterprise Agents Backend Connectors GenAI Models A2A MCP OpenAI Workflow AI Gateway
31. Agentic Layer - Integration of n8n Workflows
32. Ready to go Agentic? Stay up-to-date with the Agentic Layer Newsletter! We look forward to continuing our discussion about Agentic AI with you!
33. AI native application and system landscapes of tomorrow and thus hypothesis of how companies will change digitally Core system (with data and transaction sovereignty) Other centralised systems A core system as SaaS with low vertical integration and with high standard fidelity ("clean core") & stability. Customised extensions only via individual software. Conversational user interfaces Chat systems as primary user interfaces of employees and customers Data platform (usually data lakehouse technology plus metadata management) AI & automation platform (usually Agentic Automation, GenAI, ML and Process Mining) Development platform (usually DevOps/IDP with cloud technology) The IT becomes a platform provider, which enables business departments to create vertical solutions - while maintaining a high level of compliance, security, production maturity and operational stability. Applications Assistants Agents Customised or ready-made vertical solutions that are developed under the responsibility of business departments as AI-accelerated composable applications on underlying platforms APIs Workflows Analysis Customisations Customisations
34. QAware GmbH | Aschauer Straße 30 | 81549 München | GF: Dr. Josef Adersberger, Michael Stehnken, Michael Rohleder, Mario-Leander Reimer Niederlassungen in München, Mainz, Rosenheim, Darmstadt | +49 89 232315-0 | info@qaware.de The next step? Let's talk! Mario-Leander Reimer Managing Director, CTO mario-leander.reimer@qaware.de +49 151 61314748