Scaling AI Applications with Kubernetes ( By: Sheharyar Naseer) at Build with AI 2025

Talk by Sheharyar Naseer (https://www.linkedin.com/in/sheharyarn/) at Build with AI 2025 by GDG Lahore

Transcript

1. SCALING AI APPLICATIONS WITH KUBERNETES Google Developers Group Lahore ▪︎ September 2025
2. Outline Sheharyar Naseer Systems architect & technology advisor for startups and enterprises. Find me online @sheharyarn
3. Background ✦ Indie nomad so ware architect ✦ 15+ years of polyglot experience, focus on Web & Cloud ✦ Worked with Apple, Superlist, AllocatorOne, TheScore, Slab, etc. ✦ StackOver ow: 75,000+ score (Top 5 in Pakistan) ✦ Author / Contributor of multiple famous libraries & tools ft fl ✦ Featured on popular developer communities
4. Outline ✦ State of AI ✦ Scaling Challenges ✦ Using Kubernetes ✦ Q/A
5. State of AI What does the AI landscape look like today?
6. The AI Explosion ✦ Unprecedented growth and use of AI ✦ 72% of businesses have adopted AI ✦ 700M+ people used AI apps in [Source] rst-half 0f 2025 ✦ Market size to reach $1.34 Trillion USD by 2030 [Source] [Source] ✦ Massive investments in startups and infrastructure fi ✦ Systems need to scale to handle demand
7. AI Spectrum ✦ External API calls (Gemini, Claude, OpenAI) ✦ Self-hosted or Custom ✦ LLMs ✦ RAG Systems (Vector, Embedding, and other services) ✦ Vision Inference (Detection, Segmentation, etc.) ✦ Traditional ML (Regression, Classi cation, Clustering, Anomalies, etc.) ✦ Generative models fi ✦ Hybrid Systems
8. LLM Dominance ✦ Highest visibility ✦ Mainstream adoption ✦ Platform e ect ✦ All the hype ✦ Versatile applications Even used where it shouldn't be ff ✦
9. Scaling Challenges Common problems when scaling AI applications
10. Growing Pains ✦ Scaling systems is already hard ✦ New sets of challenges per type ✦ Vary greatly by scope ✦ No single strategy that can be applied everywhere
11. Examples ✦ Common ✦ Resource-bound operations ✦ Over/under-provisioned infrastructure ✦ Cost management ✦ External APIs (Rate-limits) ✦ LLMs (Cold-starts, Scheduling, Fragmentation) ✦ RAGs (Data consistency, Pipeline coordination, Storage) ✦ Hybrid Systems (Coordination + all of the above)
12. Kubernetes for Scale How can Kubernetes help us scale AI applications
13. What is K8S? ✦ Open-source container automation framework ✦ Simple, modular and declarative ✦ Automated: Monitors, detects & recti es issues ✦ Primitives ✦ Pods, Services, Volumes, Operators, Scheduling, etc. ✦ Focus External API AI applications fi ✦
14. Scaling Strategies ✦ Vertical Pod Autoscaler (VPA) ✦ Adjusts resources within pods ✦ Horizontal Pod Autoscaler (HPA) ✦ Scale number of pods up/down ✦ K8S Event-Driven Autoscaler (KEDA) ✦ Scale number of pods up/down
15. Vertical Pod Autoscaler ✦ Adjusts CPU & memory for pods ✦ Recommender: ✦ Calculates optimal resource values by analyzing historical metrics ✦ Sets two parameters: Target utilization & min/max resources allowed ✦ Updater: ✦ Monitors recommendation changes ✦ Replaces pods when adjustments are needed
16. Horizontal Pod Autoscaler ✦ Adjusts number of pods replicas ✦ Based on workload ✦ Resource utilization (CPU/memory) ✦ Tra c-Driven ✦ Scales both up and down ffi ✦ Most common & widely used pattern
17. K8S Event-Driven Autoscaler ✦ Supports both Horizontal & Vertical scaling ✦ Based on custom events & metrics ✦ Ideal for: ✦ Event-driven architectures (message queues, event buses) ✦ Serverless-style applications (scale to zero) ✦ Workloads determined by data availability
18. More Tools & Platforms ✦ We're only scratched the surface ✦ Core Tools ✦ MLFlow ✦ Seldon Core ✦ KServe ✦ Runtimes ✦ Nvidia Triton ✦ TensorFlow Serving
19.   Questions? These Slides → shyr.io/t/scaling-ai-with-k8s More Talks → shyr.io/talks shyr.io hi@shyr.io @sheharyarn @