Leveraging Microservices and Traditional DevOps for Next-Gen Operations (GenOps)

Transcript

1. Leveraging Microservices and Traditional DevOps for Next-Gen Operations (GenOps) Jakarta Ananda Dwi Rahmawati
2. Ananda Dwi Rahmawati - Cloud and DevOps Engineer @ Singapore - Google Developer Expert Cloud - Modern Architecture - https://linktr.ee/misskecupbung Jakarta
3. What is GenOps?
4. 1. E-commerce Platform 2. Banking System 3. Content Management System (CMS) 4. Inventory Management System Traditional Applications The unit is typically a microservice, each a self-contained functional unit deployed in a container-native runtime like Kubernetes.
5. 1. Chatbot with Adaptive Responses 2. Automated Content Creation Tool 3. AI-Powered Data Analysis Tool 4. Creative Design Assistant Generative AI Applications The unit is the Generative AI Agent, which encapsulates not just functional code but additional components enabling non-deterministic processing and output.
6. Jakarta GenOps combines DevOps principles with ML workflows to deploy, monitor, and maintain Gen AI models in production. It ensures Gen AI systems are scalable, reliable, and continuously improving.
8. Scale: Billions of parameters require specialized infrastructure. Compute: High resource demands for training and inference. Safety: Need for robust safeguards against harmful content. Rapid evolution: Constant updates to keep pace with new developments. Why do we need a GenOps? Unpredictability: Non-deterministic outputs complicate testing and validation.
9. Key capabilities of GenOps
10. GenOps Elements for pre-trained and fine-tuned models Gen AI experimentation and prototyping Prompt Evaluation Safety Deployment Fine-tuning Version Control Security and governance Optimization Prompt versioning Prompt engineering Prompt enhancement
11. Key Operational Considerations Non-Deterministic Behavior Iterative Processing and Memory Use Resource Optimization
13. Model Compliance and Approval Control. Google Cloud Model Garden Model Security. Model Armor. Prompt Version Management. Model and Prompt Evaluation. Model Evaluation Services. GenOps with Google Cloud
14. Centralized Tool Management. Document AI Layout Parser, Vertex AI Vector Search & Multimodal Embeddings API, Vertex AI Search. API Management with Apigee. Apigee API Gateway. GenOps with Google Cloud
16. Few shot examples: provide examples to guide the model's output format, phrasing, scope, and pattern. Supervised fine-tuning dataset: This labeled dataset is used to fine-tune a pre-trained model to a specific task or domain. Golden evaluation dataset: To assess the performance of the model for a given task, can be used for both manual and metric-based evaluation. Data: The Generative AI journey starts with data.
17. Vertex AI Studio allows collaborative prompt creation, testing, and refinement. Prompt management
18. Model fine-tuning: adjusting the pre-trained Gen AI model to specific tasks or domains using the fine-tuning data. Supervised fine-tuning, is useful for clear tasks with labeled data, especially when the content is different from the model's original training. Reinforcement Learning from human feedback(RLHF) uses feedback gathered from humans to tune a model. Prompt Visualization using TensorBoard
19. Model fine-tuning: adjusting the pre-trained Gen AI model to specific tasks or domains using the fine-tuning data. The UI can also be created using open source solutions like Google Mesop. It helps human evaluators to evaluate and update the LLMs responses.
20. Model-based metrics use a Google model to evaluate results, either by comparing pairs or individual outputs. Computation-based metrics, like ROUGE and BLEU, use math to compare the model’s output to a reference. Model evaluation: The GenAI Evaluation Service in Vertex AI is used to evaluate GenAI models with explainable metrics. VertexAI AutoSxS Evaluation
21. Models with managed APIs, like Google’s Gemini, can accept prompts directly without deployment. Other Gen AI models must be deployed to a VertexAI Endpoint to accept prompts, e.g Fine-tuned models and Gen AI models without managed APIs, like Gemma2, Mistral, and Nemo, Model deployment
22. Ongoing monitoring assesses the deployed models' real-world performance, analyzing key metrics, data patterns, and trends. Monitoring
23. References • https://cloud.google.com/blog/products/devops-sre/genops-learnings-f rom-microservices-and-traditional-devops • https://cloud.google.com/blog/products/ai-machine-learning/learn-how- to-build-and-scale-generative-ai-solutions-with-genops • https://cloud.google.com/vertex-ai/generative-ai/docs/models/evaluatio n-overview
24. Thank you Jakarta