Build Your Own Secured AI Platform with Google Cloud Vertex AI by unleashing the Power of Multi-LLM by Eckarath Khounsombath

In today's AI-driven world, organizations face the challenge of harnessing the power of Large Language Models (LLMs) while maintaining data privacy and avoiding dependency on a single provider. This talk presents a comprehensive solution: creating your own multi-LLM, image generation service using Google Cloud Platform.

Key points covered:

- Discovering Google Vertex AI: Get hands-on Google Vertext AI solution suits to benefits of using a diverse range of LLMs, AI Services and learn how to combine their strengths to achieve superior results.

- Building Your Own AI Sanctuary/: Learn how to integrate multiple LLMs and image generation models through a unified API, ensuring your data remains under your control and avoiding vendor lock-in.

- Knowledge-Based Solutions (RAG) on Steroids: Explore how to build powerful knowledge bases that generate insightful responses and visually compelling explanations, leveraging the combined capabilities of LLMs.

- Agentic-Centric Solutions: The Future of AI Interaction: Create AI agents that understand and respond to complex prompts, generating both text and images to fulfill user requests, from creative storytelling to research assistance.

https://youtu.be/NVw_m9Nsc6w

DevFest Montreal 2024

Transcript

1. Build Your Own Secured AI Platform with Google Cloud Vertex AI by unleashing the Power of Multi-LLM MONTRÉAL Eckarath Khounsombath
2. Business & Technology Architect Eckarath Khounsombath From Internet Pioneers to Cloud Innovators and AI Trailblazers
3. The Dawn of GenAI Why ? What ? How ? What’s Next Agenda
4. The Dawn of GenAI A Modern fairytale MONTRÉAL
5. We stand at the cusp of a technological revolution—the era of generative AI. GenAI is rapidly transforming industries and redefining what's possible. GenAI is unlocking unprecedented opportunities for businesses of all sizes
6. Samsung employees pasted sensitive data into ChatGPT to help into their day-to-day work MONTRÉAL
7. Samsung employees pasted sensitive data into ChatGPT to help into their day-to-day work MONTRÉAL Unawareness of how ChatGPT handles data (it's sent to OpenAI for training).
8. Samsung employees pasted sensitive data into ChatGPT to help into their day-to-day work MONTRÉAL Unawareness of how ChatGPT handles data (it's sent to OpenAI for training). Potential compromise of confidential information, reputational damage.
9. 50 More than new LLMs models popup since ChatGPT appears
10. Every tech company is now an AI company.
11. Why ? A GenAI Governance MONTRÉAL
12. GenAI is transforming how businesses operate. To maximize benefits and minimize risks, enterprises must establish clear governance frameworks
13. 1. Vision 2. Value 3. Risk 4. Adoption Build a GenAI Strategy GenAI needs to pivot from a futuristic novelty to practical, actionable business applications. Companies needs to capitalize on its capabilities to solve real-world business problems and achieve tangible outcomes with this new technology. A GenAI strategy must anchor their approach in four fundamental pillars 4 Fundamental Pillars
14. What ? GenAI - Use Cases MONTRÉAL
15. Common Use Cases AI Content Creation AI Content Discovery AI Conversational AI Simulations Writing/Text, Speech, Audio, Image, Video, Code Search, Analysis, Knowledge Management Virtual Assistant, Translation Prediction, Synthetic Data, Simulation
16. High Level Design Infrastructure DataLake LLM Mesh Observability Private LLM Custom LLM Vector Store Security Monitoring API GenAI Apps Internal Apps GenAI Studio Reporting GenAI Services API Discovery Content Conversation
17. Google Cloud Platform brings us the perfect backbone to build our GenAI Enterprise Platform.
18. How ? Google Cloud Platform fit it all MONTRÉAL
19. Low Level Design - Compute & Data Layer Infrastructure DataLake Use to deploy complex LLM model Use to deploy simplest model Use GPU/TPU to execute models Use data services for : • Usage in GenAI • Training/Augmenting LLM
20. Low Level Design - LLM Mesh LLM Mesh Private LLM Custom LLM Vector Store API Cloud API Vertex AI Model Garden CloudRun with GPUs CloudRun
21. Low Level Design - GenAI Services GenAI Services API Discovery Content Conversation Vertex AI Agent Builder Build your own RAG Vertex AI Agent Builder Build your own Chatbot/Assistant Google Translate Google AI Service Expose secured API for general usage LLM Mesh API
22. Low Level Design - Observability Observability Security Monitoring Reporting Cloud Monitoring Cloud Logging Cloud Trace Identity & Access Management Security Command Center BigQuery Pub/Sub LLM Mesh GenAI Services Google Data Studio
23. LLM Mesh Technical Design - Multi-LLMs Vertex AI Model Garden Mistral Anthropic LLama API / SDK CloudRun LLM API CloudRun Ollama Mistral (1) Use Model Garden to let VertexAI to manage LLM models deploiement for you. You will able to access to LLM model through a VertexAI Endpoint or with VertexAI SDK if it has been embedded (1’) Use CloudRun with GPUs option to deploy Ollama and available LLM. You will be able to access to LLM model through Ollama API. (2) Call dedicated API or SDK related to the selected LLM to answer the request Observability E 1 2 1 1’ 3 4 5 (5) Send all usage of Services by logging, sending request to PubSub/BQ, tracing, … API Generic LLM inference (1) Receive a request to find an answer with a specific LLM (4) Send generic answer (3) Send an answer in selected LLM format
24. GenAI Services (1) Use Agent Builder Datastore to RAGify your data to be interrogated. Technical Design - Knowledge Base w/ Agent Builder Agent Builder DataStore Web GCS, Drive Database API CloudRun GenAI Services API Agent Builder Search App (2) Use Agent Builder Search Application to create dedicated Knowledge following a list of datastore (2)Call Vertex AI Endpoint to use search app to get an answser from request (5) Send all usage of Services by logging, sending request to PubSub/BQ, tracing, … Observability WebUI Automation AgentBuilder DataStore & Search App can be create through Cloud API Limitation AgentBuilder DataStore & Search App can be only use with Google LLMs models E 4 2 1 1 2 3 5 (3) Send a formatted answer with sources Service Knowledge Base (1) Receive a request to find an answer in a specific Knowledge Base (4) Send an enrich answer with link to the sources
25. GenAI Services LLM Mesh Technical Design - Knowledge Base w/ Langchain CloudRun GenAI Services API (1) Use Cloud Storage to store (un)structured documents for your Knowledge Base Observability CloudRun (Jobs) Embeddings CloudSQL Vector Store Cloud Storage Documents E 1 2 3 5 4 5 1 2 4 3 6 (2) Use EventArc to trigger job to ingest documents received (3) Use Cloud Run Jobs to ingest documents using Langchain framework (read, chunk, embeddings) (4) Use LLM Mesh API to create embeddings with LLM models available (5) Use CloudSQL PostgreSQL with pgvector plugin to store embeddings vector (2) Embed the request (6) Send all usage of Services by logging, sending request to PubSub/BQ, tracing, … (4) Call specific LLM to generate an answer from request + documents chunk retrieved (1) Receive a request to find an answer in a specific Knowledge Base (5) Send an enrich answer with link to the sources (3) Use embeddings to do a vector search
26. High Level Design - Assistant Anatomy 1 2 3 Goals High level description of the goal the assistant intends to accomplish Instructions Tools A step-by-step execution instructions to accomplish target goal Assistant can use selected tools (API, documents, …) to generate response Agentic Framework
27. GenAI Services (1) Use Tools connector providing by Agent Builder to empower your assistan. Technical Design - Assistant w/ Agent Builder Agent Builder DataStore API CloudRun GenAI Services API Agent Builder Agent (2) Use Agent Builder Agent Application to create dedicated Knowledge following a list of datastore (2)Call Vertex AI Endpoint to use search app to get an answser from request (5) Send all usage of Services by logging, sending request to PubSub/BQ, tracing, … Observability WebUI Automation AgentBuilder Agent can be create through Cloud API Limitation AgentBuilder DataStore & Search App can be only use with Google LLMs models E 4 2 1 2 3 5 (3) Send a formatted answer with sources Service Knowledge Base (1) Receive a request to find an answer in a specific Knowledge Base (4) Send an enrich answer with link to the sources Function Calling Tools OpenAPI 1
28. What’s Next ? From Passive Tools to Proactive Agents MONTRÉAL
29. ( 1 ) Key Takeaways GenAI Governance Establish governance frameworks to maximize GenAI benefits and minimize risks. ( 2 ) GenAI Platform Use a Multi-LLM approach to package GenAI services to be served in an API first paradigm ( 3 ) Google Cloud for GenAI Platform Leverage GCP services like Vertex AI, CloudRun, and LangChain to create a multi-LLM, knowledge-driven GenAI platform.
30. ( 1 ) Towards Agentic AI GenAI (Generative AI) has rapidly evolved from narrow, constrained models to large, flexible language models. ( 2 ) These models are demonstrating increasingly advanced language understanding and generation capabilities. ( 3 ) But GenAI is poised to become even more "agentic" - gaining the ability to reason, plan, and take autonomous actions.
31. REMEMBER Every prompt we send creates a carbon footprint. Let's train our AIs to be smarter, not just bigger, and make every interaction count for a sustainable digital future.