Reconciling High Accuracy, Cost-Efficiency, and Low Latency of Inference Serving Systems

Transcript

1. Reconciling High Accuracy, Cost-Ef f iciency, and Low Latency of Inference Serving Systems Pooyan Jamshidi University of South Carolina
2. 2 Teamwork
3. 2 Teamwork
4. 2 Teamwork
5. 2 Teamwork
6. Outline 3 InfAdapter IPA Background
7. Multi-objective performance tradeoff
8. 5 Research Production Objectives Model performance* Different stakeholders have different objectives “*” It’s actively being worked. See Utility is in the Eye of the User: A Critique of NLP Leaderboards (Ethayarajh and Jurafsky, EMNLP 2020) ML in research vs. in production
9. 6 ML team highest accuracy Stakeholder objectives
10. 7 ML team highest accuracy Sales sells more ads Stakeholder objectives
11. 8 ML team highest accuracy Sales sells more ads Stakeholder objectives Product fastest inference
12. 9 ML team highest accuracy Sales sells more ads Manager maximizes profit = laying off ML teams Stakeholder objectives Product fastest inference
13. 10 Research Production Objectives Model performance Different stakeholders have different objectives Computational priority Fast training, high throughput Fast inference, low latency Computational priority generating predictions
14. Latency matters Latency 100 -> 400 ms reduces searches 0.2% - 0.6% (2009) 30% increase in latency costs 0.5% conversion rate (2019) 11
15. 12 • Latency: time to move a leaf • Throughput: how many leaves in 1 sec
16. 13 • Real-time: low latency = high throughput • Batched: high latency, high throughput
17. ML Serving
18. System = Software + Middleware + Hardware CPU Memory Controller GPU Lib API Clients Devices Network Task Scheduler Device Drivers File System Compilers Memory Manager Process Manager Frontend Application Layer OS/Kernel Layer Hardware Layer Deployment SoC Generic hardware Production Servers
19. Model Serving Abstract level
20. Model Serving TF Serving
21. Model Serving Web app
22. Model Serving Internet of Thing
23. Model Serving Stream Processing System
24. Model Serving Pipeline
25. InfAdapter
26. EuroMLSys ’23, May 8, 2023, Rome, Italy 23
27. “More than 90% of data center compute for ML workload, is used by inference services” 24
28. ML inference services have strict requirements 25 Highly Responsive!
29. ML inference services have strict requirements 26 Highly Responsive! Cost-Efficient!
30. ML inference services have strict requirements 27 Highly Accurate! Highly Responsive! Cost-Efficient!
31. ML inference services have strict & conflicting requirements 28 Highly Accurate! Highly Responsive! Cost-Efficient!
32. More challenge: Dynamic workload 29
33. Existing adaptation mechanisms 30 Resource Scaling Vertical Scaling (AutoPilot EuroSys’20) Horizontal Scaling (MArk ATC’19) Quality Adaptation Multi Variants (Model-Switching Hotcloud’20)
34. Resource allocation 31 Over Provisioning Under Provisioning
35. Resource allocation 32
36. Resource allocation 33
37. Resource allocation 34
38. Resource allocation 35
39. Resource allocation 36
40. Quality adaptation 37 ResNet18: Tiger ResNet152: Dog
41. Quality adaptation 38
42. Solution: InfAdapter InfAdapter is a latency SLO-aware, highly accurate, and cost-efficient inference serving system. 39
43. InfAdapter: Why? Different throughputs with different model variants 40
44. InfAdapter: Why? Higher average accuracy by using multiple model variants 41
45. InfAdapter: How? Selecting a subset of model variants, each having its own size Meeting latency requirement for the predicted workload while maximizing accuracy and minimizing cost 42
46. InfAdapter: Design 43
47. InfAdapter: Design 44
48. InfAdapter: Formulation 45
49. InfAdapter: Formulation 46 Maximizing Average Accuracy
50. InfAdapter: Formulation 47 Maximizing Average Accuracy Minimizing Resource and Loading Costs
51. InfAdapter: Formulation 48
52. InfAdapter: Formulation 49 Supporting incoming workload
53. InfAdapter: Formulation 50 Supporting incoming workload Guaranteeing end-to-end latency
54. InfAdapter: Design 51
55. InfAdapter: Experimental evaluation setup Twitter-trace sample (2022-08) Baselines Kubernetes VPA and adapted Model-Switching Used models Resnet18, Resnet34, Resnet50, Resnet101, Resnet152 Interval adaptation 30 seconds A Kubernetes cluster of 2 computing nodes 48 Cores, 192 GiB RAM 52
56. Workload Pattern 53
57. InfAdapter: P99-Latency evaluation 54
58. InfAdapter: P99-Latency evaluation 55
59. InfAdapter: P99-Latency evaluation 56
60. InfAdapter: P99-Latency evaluation 57
61. InfAdapter: P99-Latency evaluation 58
62. InfAdapter: P99-Latency evaluation 59
63. InfAdapter: P99-Latency evaluation 60
64. InfAdapter: P99-Latency evaluation 61
65. InfAdapter: Accuracy evaluation 62
66. 63 InfAdapter: Cost evaluation
67. InfAdapter: Experimental evaluation 64 Compare aggregated metrics of latency SLO violation, accuracy and cost with other works on different β values to see how they perform on different accuracy-cost trade-off
68. Takeaway 65 Inference Serving Systems should consider accuracy, latency, and cost at the same time.
69. Takeaway 66 Model variants provide the opportunity to reduce resource costs while adapting to the dynamic workload. Using a set of model variants simultaneously provides higher average accuracy compared to having one variant. Inference Serving Systems should consider accuracy, latency, and cost at the same time.
70. Takeaway 67 Model variants provide the opportunity to reduce resource costs while adapting to the dynamic workload. Using a set of model variants simultaneously provides higher average accuracy compared to having one variant. Inference Serving Systems should consider accuracy, latency, and cost at the same time. InfAdapter!
71. 68 https://github.com/reconfigurable-ml-pipeline/InfAdapter
72. IPA
74. Inference Pipeline Recommender Systems Source: https://developer.nvidia.com/blog/ optimizing-dlrm-on-nvidia-gpus/ Video Pipelines Source: https://docs.nvidia.com/metropolis/ deepstream/5.0/dev-guide/index.html#page/ DeepStream_Development_Guide/ deepstream_overview.html
75. 72 Autoscaling Previous works have used auto scaling for cost optimization of inference pipeline
76. Is only scaling enough? ?
77. Effect of Batching
78. How to navigate Accuracy/latency trade off? Model Variants and Model Switching! Previous works INFaaS and Model-Switch have proven that there is a big a latency-accuracy- resource footprint tradeoffs of models trained for the same task
79. How to navigate Accuracy/latency trade off? Model Variants and Model Switching! Previous works INFaaS and Model-Switch have proven that there is a big a latency-accuracy- resource footprint tradeoffs of models trained for the same task
80. How to navigate Accuracy/latency trade off? Model Variants and Model Switching! Previous works INFaaS and Model-Switch have proven that there is a big a latency-accuracy- resource footprint tradeoffs of models trained for the same task
81. Search Space
82. 77 Goal: Providing a flexible inference pipeline
83. 78 Snapshot of the System
84. 79 System Design
85. Problem Formulation Objective function Accuracy Objective Resource Objective Batch Control Latency SLA Throughput Constraint One active model per node
86. Implementation and Experimental
 Setup 81
87. 1. Industry standard 2. Used in recent research 3. Complete set of autoscaling, scheduling, observability tools (e.g. CPU usage) 4. APIs for changing the current AutoScaling algorithms 1. Industry standard ML server 2. Have the ability make inference graph 3. Rest and GRPC endpoints 4. Have many of the features we need like monitoring stack out of the box How to navigate Model Variants
88. 83 Experimental Setup • A six node Kubernetes cluster
89. Experimental Results 84
90. 85 Video Pipeline
91. 86 Audio + QA Pipeline
92. 87 Summarization + QA
 Pipeline
93. 88 Summarization + QA
 Pipeline
94. 89 NLP Pipeline
95. 90 Adaptivity to multiple objectives
96. 91 Effect of predictor
97. 92 Gurobi solver scalability
98. Model Serving Pipeline https://github.com/reconfigurable-ml-pipeline/ipa
99. Model Serving Pipeline Is only scaling enough? ? https://github.com/reconfigurable-ml-pipeline/ipa
100. Model Serving Pipeline Is only scaling enough? ? X Snapshot of the System https://github.com/reconfigurable-ml-pipeline/ipa
101. Model Serving Pipeline Is only scaling enough? ? X Snapshot of the System X Adaptivity to multiple objectives https://github.com/reconfigurable-ml-pipeline/ipa