AI in the Real World: An Introductory Guide to Building & Deploying AI Systems

Pijak in collaboration with IBM SkillsBuild
2 April 2026
https://www.youtube.com/live/fX1pwJ1Ryps?si=th4S_VEOmgjtrA2K

Transcript

1. AI in the Real World: An Introductory Guide to Building & Deploying AI Systems Kuncahyo Setyo Nugroho | NVIDIA AI R&D Center - BINUS University Prepared & Presented by Kuncahyo Setyo Nugroho © 2026
2. Prepared & Presented by Kuncahyo Setyo Nugroho © 2026
3. Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Read more: https://www.linkedin.com/pulse/630-billion-question-why-80-ai-projects-fail-brian-will-psile
4. The Five Root Causes: A Framework Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 ▪ Unclear Problem The project starts without a clear business problem to solve. ▪ Bad or Unready Data The data is messy, incomplete, or not suitable for AI. ▪ People Don’t Use It Users don’t trust or adopt the AI system. ▪ Weak Infrastructure The infrastructure is not strong enough for real-world use. ▪ Can't Scale to Production The project works in testing but fails when scaled to real use. AI Product Failure ≠ Model Failure “AI doesn’t fail because of the model — it fails because everything around the model is not ready”
5. The AI Lifecycle You Probably Know Define Problem Collect Data Clean & Annotate Data Train Model Evaluate / Test Model Communicate Results “Flat-earth” AI Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 This is how most people think AI works
6. But Building AI Products is Different Define Problem Collect Data Clean & Annotate Data Train Model Evaluate / Test Model Deploy Model Monitor System Communicate Results How AI Systems (Product) Are Built Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 The real challenge starts AFTER the model works
7. Where Does AI Fit in a Real System? Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Client Server Database Local Remote Request Response The key question: Where should the AI model live? In most real-world applications, systems follow a client–server architecture
8. Do AI systems always need to be real-time? Prepared & Presented by Kuncahyo Setyo Nugroho © 2026
9. Option 1: Batch Prediction Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Client Server Database Model Model runs on schedule. Results stored in database before users even ask. Local Remote Request Response Pre-computed predictions
10. Option 1: Batch Prediction (Pros & Cons) Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Pros Cons ▪ Simple to implement ▪ Scales easily. Just run on more data ▪ Fast for users. Prediction already ready ▪ Battle-tested in large-scale systems for years ▪ Not real-time. Users get "stale" predictions ▪ Doesn't handle unpredictable user inputs well ▪ Hard to detect when model becomes outdated
11. What if we need real-time predictions? Prepared & Presented by Kuncahyo Setyo Nugroho © 2026
12. Option 2: Model-in-Service Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Client Server Database Model Model “lives” inside the server. Simple but creates tight coupling. Local Remote Request Response
13. Option 2: Model-in-Service (Pros & Cons) Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Pros Cons ▪ Simple architecture ▪ Re-uses existing infrastructure ▪ Good starting point for small applications ▪ Web server may be written in a different language than the model ▪ Large models “eat” into server resources ▪ Model and server scale differently ▪ Model updates require redeploying the whole app
14. What if our system needs to scale? Should the model and the app scale together? Prepared & Presented by Kuncahyo Setyo Nugroho © 2026
15. Option 3: Model-as-Service Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Client Server Database Model Model runs independently. Can be reused by multiple apps via API. Local Remote Request Response Model Service (It’s own server)
16. Building a Model Service: REST APIs Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Method Request POST /predict { "size": 120, "rooms": 3 } Transform ([120, 3]) Serving predictions in response to canonically-formatted HTTP requests. Why API Matters: ▪ Any app can “talk” to the model ▪ Model and app can be built in different languages ▪ Easy to update the model without changing the app, vice versa Integration Request Client Endpoint (Model Service) Integration Response Method Response model.predict ([120, 3]) = 850_000_000 { "predict": 850000000 } { "price": "Rp 850 jt" }
17. Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Constraining Model Dependencies: ONNX The Promise: Define your model in any framework, run it consistently anywhere. The Reality: Framework (library) change fast, bugs in the translation layer are common, and not all operations are supported.
18. Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Managing Dependencies with Containers: Docker
19. Option 3: Model-as-Service (Pros & Cons) Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Pros Cons ▪ Model bugs won't crash the main app ▪ Scale model independently from the app ▪ One model can serve multiple apps ▪ Update model without touching the main app ▪ Can add latency. Extra network round trip ▪ Adds infrastructure complexity ▪ Need to manage a separate model service
20. What if we can’t rely on the network? Prepared & Presented by Kuncahyo Setyo Nugroho © 2026
21. Option 4: Edge Prediction Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Client Server Database Model Local Remote Request Response Model runs directly on the user's device. No server call needed for inference. No internet needed
22. Option 4: Edge Prediction (Pros & Cons) Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Pros Cons ▪ Lowest latency. No network round trip ▪ Works without internet connection ▪ Data never leaves the device ▪ Each device runs its own model ▪ Limited hardware on user's device ▪ Mobile frameworks are less powerful ▪ Difficult to update models ▪ Hard to monitor and debug in production
23. Train, Test, Deploy. You're Done... Right? 🤔" Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 ▪ Validation loss is below your target performance ▪ Test loss is not much worse than validation ▪ Model performs well across all critical slices and metrics ▪ Qualitatively the predictions make sense ▪ You verified that the prod model has the same performance as the dev model ▪ You verified that the prod model is indeed better than the previous one
24. The Dream of How This Would Work Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 If Everything Goes Right More User 01 More Data 02 Better Model 03 Better Product 04 But this only works if you know what's happening after deploy!
25. Your AI system is live. Now the real work begins. Because the real world keeps changing… Your system must be monitored! Prepared & Presented by Kuncahyo Setyo Nugroho © 2026
26. Business Metrics (e.g., revenue, conversion rate) What to Monitor? Practical Recommendations Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 More Informative Less Informative Feasibility of Measurement Value Easier Hard Model Metrics (e.g., accuracy, F1-score) Model Input & Prediction (e.g., prediction output) User Feedback (e.g., rating) System (App) Performance (e.g., latency, CPU usage)
27. How to Think About Continual Learning Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 Logging Curation Retraining Trigger Offline Testing Deployment Re-Training Dataset Preparation Request Response (Results) Feedback App (System) User What data should we store from user interactions? Which data is most valuable for improving the model? When should we retrain the model? How do we know the model works in the real world? What does “good enough” look like for all stakeholders? Are we actually improving the model? Does our data reflect the real-world problem? AI Engineer (Tune the Strategy/ Monitor Metrics)
28. Prepared & Presented by Kuncahyo Setyo Nugroho © 2026
29. Most Common AI/ML Roles Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 High Low AI/ML Skills Software Engineering Skills Low High AI/ML Product Manager AI/ML Researcher Data Scientist AI/ML Engineer MLOps / Infra Defines problems and aligns AI solutions with business needs Size of bubble = communication / technical writing Extracts insights and builds models from data Builds systems to deploy and scale models Builds reliable, production-ready AI systems Develops new algorithms and advances AI capabilities Connects systems, data, and models
30. How to Stand Out in AI/ML Roles Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 ▪ Show genuine interest in ML e.g., attend conferences, complete online courses/workshop ▪ Build strong software engineering skills e.g., focus on writing clean and scalable code, understanding systems beyond just modeling ▪ Demonstrate broad AI/ML knowledge e.g., Write and share insights from ML projects or research in your own words ▪ Prove you can deliver real projects e.g., build side projects end-to-end solutions, not just notebooks ▪ Show creative AI/ML thinking to solve real-world problems e.g., join Kaggle competitions, publish research (journal/conference) Higher Impact
31. Key Takeaways Prepared & Presented by Kuncahyo Setyo Nugroho © 2026 ▪ AI product success is not about the model, but about the system around it ▪ Choosing where the model lives is a key architectural decision that affects performance, scalability, and reliability ▪ Real-world AI systems require more than training, they require deployment, monitoring, and iteration ▪ Without monitoring, even a good model will fail in production due to changing real-world data ▪ Continual learning is essential to keep the model relevant and improving over time ▪ Building AI systems requires combining data, engineering, and infrastructure, not just modeling skills ▪ The most valuable role is the one that can connect models, systems, and real-world impact
32. Thank You https://www.instagram.com/ksnugroho https://www.linkedin.com/in/ksnugroho ksnugroho26@gmail.com kuncahyo.nugroho@binus.edu Get in touch