Build Your Own Secured AI Platform with Google Cloud Vertex AI by unleashing the Power of Multi-LLM MONTRÉAL Eckarath Khounsombath

Business & Technology Architect Eckarath Khounsombath From Internet Pioneers to Cloud Innovators and AI Trailblazers

The Dawn of GenAI Why ? What ? How ? What’s Next Agenda

The Dawn of GenAI A Modern fairytale MONTRÉAL

We stand at the cusp of a technological revolution—the era of generative AI. GenAI is rapidly transforming industries and redefining what's possible. GenAI is unlocking unprecedented opportunities for businesses of all sizes

Samsung employees pasted sensitive data into ChatGPT to help into their day-to-day work MONTRÉAL

Samsung employees pasted sensitive data into ChatGPT to help into their day-to-day work MONTRÉAL Unawareness of how ChatGPT handles data (it's sent to OpenAI for training).

Samsung employees pasted sensitive data into ChatGPT to help into their day-to-day work MONTRÉAL Unawareness of how ChatGPT handles data (it's sent to OpenAI for training). Potential compromise of confidential information, reputational damage.

50 More than new LLMs models popup since ChatGPT appears

Every tech company is now an AI company.

Why ? A GenAI Governance MONTRÉAL

GenAI is transforming how businesses operate. To maximize benefits and minimize risks, enterprises must establish clear governance frameworks

1. Vision 2. Value 3. Risk 4. Adoption Build a GenAI Strategy GenAI needs to pivot from a futuristic novelty to practical, actionable business applications. Companies needs to capitalize on its capabilities to solve real-world business problems and achieve tangible outcomes with this new technology. A GenAI strategy must anchor their approach in four fundamental pillars 4 Fundamental Pillars

What ? GenAI - Use Cases MONTRÉAL

Common Use Cases AI Content Creation AI Content Discovery AI Conversational AI Simulations Writing/Text, Speech, Audio, Image, Video, Code Search, Analysis, Knowledge Management Virtual Assistant, Translation Prediction, Synthetic Data, Simulation

High Level Design Infrastructure DataLake LLM Mesh Observability Private LLM Custom LLM Vector Store Security Monitoring API GenAI Apps Internal Apps GenAI Studio Reporting GenAI Services API Discovery Content Conversation

Google Cloud Platform brings us the perfect backbone to build our GenAI Enterprise Platform.

How ? Google Cloud Platform fit it all MONTRÉAL

Low Level Design - Compute & Data Layer Infrastructure DataLake Use to deploy complex LLM model Use to deploy simplest model Use GPU/TPU to execute models Use data services for : ● Usage in GenAI ● Training/Augmenting LLM

Low Level Design - LLM Mesh LLM Mesh Private LLM Custom LLM Vector Store API Cloud API Vertex AI Model Garden CloudRun with GPUs CloudRun

Low Level Design - GenAI Services GenAI Services API Discovery Content Conversation Vertex AI Agent Builder Build your own RAG Vertex AI Agent Builder Build your own Chatbot/Assistant Google Translate Google AI Service Expose secured API for general usage LLM Mesh API

Low Level Design - Observability Observability Security Monitoring Reporting Cloud Monitoring Cloud Logging Cloud Trace Identity & Access Management Security Command Center BigQuery Pub/Sub LLM Mesh GenAI Services Google Data Studio

LLM Mesh Technical Design - Multi-LLMs Vertex AI Model Garden Mistral Anthropic LLama API / SDK CloudRun LLM API CloudRun Ollama Mistral (1) Use Model Garden to let VertexAI to manage LLM models deploiement for you. You will able to access to LLM model through a VertexAI Endpoint or with VertexAI SDK if it has been embedded (1’) Use CloudRun with GPUs option to deploy Ollama and available LLM. You will be able to access to LLM model through Ollama API. (2) Call dedicated API or SDK related to the selected LLM to answer the request Observability E 1 2 1 1’ 3 4 5 (5) Send all usage of Services by logging, sending request to PubSub/BQ, tracing, … API Generic LLM inference (1) Receive a request to find an answer with a specific LLM (4) Send generic answer (3) Send an answer in selected LLM format

GenAI Services (1) Use Agent Builder Datastore to RAGify your data to be interrogated. Technical Design - Knowledge Base w/ Agent Builder Agent Builder DataStore Web GCS, Drive Database API CloudRun GenAI Services API Agent Builder Search App (2) Use Agent Builder Search Application to create dedicated Knowledge following a list of datastore (2)Call Vertex AI Endpoint to use search app to get an answser from request (5) Send all usage of Services by logging, sending request to PubSub/BQ, tracing, … Observability WebUI Automation AgentBuilder DataStore & Search App can be create through Cloud API Limitation AgentBuilder DataStore & Search App can be only use with Google LLMs models E 4 2 1 1 2 3 5 (3) Send a formatted answer with sources Service Knowledge Base (1) Receive a request to find an answer in a specific Knowledge Base (4) Send an enrich answer with link to the sources

GenAI Services LLM Mesh Technical Design - Knowledge Base w/ Langchain CloudRun GenAI Services API (1) Use Cloud Storage to store (un)structured documents for your Knowledge Base Observability CloudRun (Jobs) Embeddings CloudSQL Vector Store Cloud Storage Documents E 1 2 3 5 4 5 1 2 4 3 6 (2) Use EventArc to trigger job to ingest documents received (3) Use Cloud Run Jobs to ingest documents using Langchain framework (read, chunk, embeddings) (4) Use LLM Mesh API to create embeddings with LLM models available (5) Use CloudSQL PostgreSQL with pgvector plugin to store embeddings vector (2) Embed the request (6) Send all usage of Services by logging, sending request to PubSub/BQ, tracing, … (4) Call specific LLM to generate an answer from request + documents chunk retrieved (1) Receive a request to find an answer in a specific Knowledge Base (5) Send an enrich answer with link to the sources (3) Use embeddings to do a vector search

High Level Design - Assistant Anatomy 1 2 3 Goals High level description of the goal the assistant intends to accomplish Instructions Tools A step-by-step execution instructions to accomplish target goal Assistant can use selected tools (API, documents, …) to generate response Agentic Framework

GenAI Services (1) Use Tools connector providing by Agent Builder to empower your assistan. Technical Design - Assistant w/ Agent Builder Agent Builder DataStore API CloudRun GenAI Services API Agent Builder Agent (2) Use Agent Builder Agent Application to create dedicated Knowledge following a list of datastore (2)Call Vertex AI Endpoint to use search app to get an answser from request (5) Send all usage of Services by logging, sending request to PubSub/BQ, tracing, … Observability WebUI Automation AgentBuilder Agent can be create through Cloud API Limitation AgentBuilder DataStore & Search App can be only use with Google LLMs models E 4 2 1 2 3 5 (3) Send a formatted answer with sources Service Knowledge Base (1) Receive a request to find an answer in a specific Knowledge Base (4) Send an enrich answer with link to the sources Function Calling Tools OpenAPI 1

What’s Next ? From Passive Tools to Proactive Agents MONTRÉAL

( 1 ) Key Takeaways GenAI Governance Establish governance frameworks to maximize GenAI benefits and minimize risks. ( 2 ) GenAI Platform Use a Multi-LLM approach to package GenAI services to be served in an API first paradigm ( 3 ) Google Cloud for GenAI Platform Leverage GCP services like Vertex AI, CloudRun, and LangChain to create a multi-LLM, knowledge-driven GenAI platform.

( 1 ) Towards Agentic AI GenAI (Generative AI) has rapidly evolved from narrow, constrained models to large, flexible language models. ( 2 ) These models are demonstrating increasingly advanced language understanding and generation capabilities. ( 3 ) But GenAI is poised to become even more "agentic" - gaining the ability to reason, plan, and take autonomous actions.

REMEMBER Every prompt we send creates a carbon footprint. Let's train our AIs to be smarter, not just bigger, and make every interaction count for a sustainable digital future.