Serverless Inferences

There are several ways to host a machine learning model on AWS. It can range from a dedicated EC2 instance to a fully managed lambda based solution. This presentation talks about several ways to host machine learning models embracing the serverless paradigm on AWS.

1. Serverless Inferences On AWS
2. What are we talking about? 2
3. 3 Define Collect Data Clean Data Build Model Train Model Deploy Model Monitor/ Operate Model
4. Pre-trained Model Model artifacts (structure, helper files, parameters, weights, ...) JSON or Binary format Can be serialized (saved) and de-serialized (loaded) 4
5. 5 Serverless!
6. 6 Inference!
7. 1. Generic Architecture What should a good design look like 7
8. 8 Load Balancer Model Server Model Artifacts
9. 2. Lambda and S3 buckets.. 9
10. 10 API Gateway Kinesis Streams Where does the pre-trained model live • Part of the lambda deployment • On demand from S3 Lambda S3
11. Trade Offs Size of the Model Memory for a Lambda Processing power for a Lambda Multiple Models? Latency of transfer from S3 11
12. 2. Sagemaker 12
13. 13 Sagemaker ECR S3 Python SDK
14. What does the sage do? 14
15. Sagemaker What does the Sagemaker do? ◍ Runs the Inference code ◍ Copies Models from S3 ◍ Load Balancer ◍ Monitoring ◍ C-test on new Models What does the Sagemaker need? ◍ Inference Code as ECR ◍ Location of Models on S3 ◍ Type of Machine to Run 15
16. Specifications 16 /opt/ml/model } The directory to which the models are copied from S3 docker run <image> serve } Sagemaker run command /ping /invocations } Endpoints that must be defined
17. 17 Sagemaker ECR S3 Python SDK /ping /invocations Where the models live
18. Inference Code 18
19. Incoming requests Reverse Proxy Inference /ping /invocations
20. Thanks! Any questions? 20