Scaling AI on a Budget: A Startup's GPU Optimization Journey by Shannon Lal

Designstripe, led by serial entrepreneur François Arbour, revolutionizes AI-powered design. Designstripe’s CTO Shannon Lal reveals how they slashed cloud costs by 30% and boosted scaling speed by 25% for GPU-intensive features.

Learn GCP optimizations applicable to startups and developers alike.

https://youtu.be/4q_fiJ3e9js

DevFest Montreal 2024

Transcript

1. Scaling AI on a Budget: A Startup’s GPU Optimization Journey
2. Agenda - Quick Intro - Steps to reduce costs and speed - Next Steps
3. Elephant in the room
4. Bio - Role: CTO of Designstripe - Previous Experience: (RBC, InVue (acquired)) - Tech Stack: NestJS/NextJS, Langchain, and Mongo - Other: Married, Father and unpaid Uber driver
5. Designstripe - Designstripe is a smart design platform, like a combination of Canva and ChatGPT, that creates ready-to-publish social media content, customizable and pre-branded for your business - Key Features: - Social Post - 3D Mockups
6. AI Models and GPU
7. Architecture Overview
8. Poll 1 Got too excited doing the Macarena and elbowed myself Mistook the closet door for the bathroom door at 3 am and got a black eye
9. Optimization Steps - HPA with PubSub - Docker Optimization - Image Streaming - Separate Nodes for Environments - Multiple Workloads
10. HPA with PubSub - Horizontal Pod Autoscaler (HPA) - Configure a custom metric on your Google Workload (deployment) - External Metric we use: - Subscription num_undelivered_messages - Target Total Value = 1
11. Docker Optimization Docker Optimizations - Remove unnecessary build files - Multi-state build Container Registry - Migrated to Artifact Registry - Single Region Artifact Artifact Registry
12. Poll 2 Son’s aim with his XBox controller has gotten a lot better Tried to prove to my spouse that I could still do a cartwheel
13. Artifact Registry - Image Streaming - Mounts the container data layer - Starts container without full image - Streams the image on demand - Leverage Multi-level caching
14. Nodes Pools - Single GPU per Node - Nvidia CUDA Image is 5GB - Model weights can get huge - Disk Pressure will be a factor - Need to separate your Node Pools by workload
15. Multiple Workloads - Reduced our memory consumption to have 2 renders on 1 GPU - LLMs can run multiple workloads on a single GPU (nvidia-smi) - Multi-Instance GPU on GKE
16. Performance Results Test Scenario Results HPA Optimization 6 minutes Image Optimization 4 minutes Pre-Loaded Nodes 2 minutes Manual Autoscale with Nodes 10 seconds Costs reduction of 40% Speed Improvement: 30 %
17. Next Steps - CloudRun with GPU - Baking Nvidia base image into node - Optimistic Loading of Nodes
18. Big Reveal
19. Thank you Contact Information: - https://www.linkedin.com/in/shannonlal/ - https://calendly.com/shannonlal/30min -
20. References HPA Autoscaler: https://cloud.google.com/kubernetes-engine/docs/samples/container-pubsub-horizontal-pod-autoscaler Docker Optimization: https://cloud.google.com/blog/products/application-development/understanding-artifact-registry-vs-container-registry https://cloud.google.com/artifact-registry/docs/repositories/repo-locations#:~:text=You%20can%20create%20repositories%20in,two%2 0or%20more%20geographic%20places Image Streaming: https://cloud.google.com/blog/products/containers-kubernetes/introducing-container-image-streaming-in-gke Multiple LLMs on single GPU: https://dev.to/shannonlal/running-multiple-llms-on-a-single-gpu-255o