Tweet
Tweet

Slide 1

Slide 1 text

Machine Learning as a Service An Entertaining Comparison by Sandeep Parikh

Slide 2

Slide 2 text

Agenda Who offers machine learning services? How do the services work? How do you use it in your applications? What we’ll cover

Slide 3

Slide 3 text

Hi, I’m Sandeep I’m a Solutions Architect at Google, working on Google Cloud Platform; I spend my time designing and documenting architectural patterns and solutions. Before that I worked for MongoDB. Before that...well a bunch of other places. I’ve been in Austin for about 12 years so I get to complain about everything. @crcsmnky

Slide 4

Slide 4 text

Just To Be Clear I am not a data scientist This talk is not sanctioned by my employer I will do my best to be unbiased

Slide 5

Slide 5 text

The Contenders Amazon Web Services Google Cloud Platform Microsoft Azure

Slide 6

Slide 6 text

Required Reading Thanks to Inês Almeida [1] for putting together an incredibly detailed and thorough Machine Learning Service Comparison [2]. She covers: ● Data sourcing ● Data preprocessing ● Model building ● Model evaluation [1] https://blog.onliquid.com/author/isbalmeida/ [2] https://blog.onliquid.com/machine-learning-service-benchmark/

Slide 7

Slide 7 text

Developer Centric Approach Not all who need machine learning are data scientists Could be used to bootstrap a larger effort Perhaps other challenges take precedence

Slide 8

Slide 8 text

Example Application Browse and rate movies Using the Movielens 10M dataset [1] Thanks to the fine folks at Grouplens [2] [1] https://movielens.org/ [2] http://grouplens.org/

Slide 9

Slide 9 text

No content

Slide 10

Slide 10 text

Code and Components https://github.com/crcsmnky/movieweb Built using Python, Flask, MongoDB Updated to use machine learning real-time endpoints from each service Will push updated code soon™

Slide 11

Slide 11 text

What’s Missing From MovieWeb? How do you know if a user will like a particular movie?

Slide 12

Slide 12 text

Steps and Goals Use Movielens training data Contains userId, movieId, rating Train a model Evaluate model Create “recommendation” service Profit!

Slide 13

Slide 13 text

AWS

Slide 14

Slide 14 text

AWS: Create Datasource ratings.csv S3

Slide 15

Slide 15 text

AWS: Create Model

Slide 16

Slide 16 text

AWS: Correcting Model Type

Slide 17

Slide 17 text

AWS: Schema configuration By default schema read in as: userId (numeric), movieId (numeric), rating (numeric) But I need: userId (numeric), movieId (numeric), rating (categorical)

Slide 18

Slide 18 text

AWS: Create Model

Slide 19

Slide 19 text

AWS: Evaluate Model 1-Click (patented!)

Slide 20

Slide 20 text

AWS: Predictions Generate real-time predictions endpoint Supply userId, movieId Returns predicted rating Upload dataset and run batch predictions One time, but need to generate dataset (about 8 GB) Repeat as-needed (daily? weekly?)

Slide 21

Slide 21 text

AWS: Limits

Slide 22

Slide 22 text

Azure

Slide 23

Slide 23 text

Azure: Create Experiment Create new datasource and upload ratings.csv Then jump into Azure ML Studio...wow! After some trial and error, landed on an approach

Slide 24

Slide 24 text

Azure: Studio

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Azure: Training Experiment choose features split 70/30

Slide 27

Slide 27 text

Azure: Evaluation Results

Slide 28

Slide 28 text

Azure: Convert to Predictive Experiment First, run training experiment Then, you’ve got a predictive experiment

Slide 29

Slide 29 text

Azure: Predictions Generate real-time predictions endpoint Supply userId, movieId Returns predicted rating (and probabilities of other labels) Upload dataset and run batch predictions One time, but need to generate dataset Repeat as-needed (daily? weekly?)

Slide 30

Slide 30 text

Azure: Limits 20 requests per endpoint (up to 10,000 endpoints) Up to 10 GB for training data No mention of batch limitations or daily limits

Slide 31

Slide 31 text

Google Cloud Platform

Slide 32

Slide 32 text

Google: Upload Data Create a Cloud Storage bucket and upload ratings.csv ...Then it’s all API-driven But, it does support Predictive Modeling Markup Language for metadata, transformations, etc.

Slide 33

Slide 33 text

Google: Train API

Slide 34

Slide 34 text

Model training is asynchronous, so you have to check when it’s ready Google: Model Availability

Slide 35

Slide 35 text

Google: Real-time Predictions Once model is trained and available, send individual API requests to predict rating Batch predictions available, done by batching API requests

Slide 36

Slide 36 text

Google: Real-time Predictions Using real-time predictions API Supply userId, movieId Returns predicted rating (and probabilities of other labels)

Slide 37

Slide 37 text

Google: Limits 2,000,000 predictions per day Training data up to 2.5 GB

Slide 38

Slide 38 text

Wrap Up

Slide 39

Slide 39 text

Takeaways AWS has a basic but powerful interface - supports regression or multiclass classification and easy model evaluation. Azure has crazy robust Studio interface with lots of algorithms and power - including ability to use BYO Python or R code Google has a limited interface and everything is opaque but supports standard PMML for metadata and transformation. Missing quality UI and scoring/eval are all behind the scenes. Marketing would tout this as “simple”

Slide 40

Slide 40 text

Takeaways AWS and Azure have support for batch predictions, which would be useful for this application Azure’s algorithm and code support is top-notch. Generates code for C#, Python, and R for batch and streaming endpoints Google requires you to manually batch but you can update your trained model with more ratings come in, reducing re-training time

Slide 41

Slide 41 text

But Wait, There’s More I’ve made a huge mistake! My training dataset had userId and movieId - those aren’t nearly enough to predict using multiclass regression Training data should have included userId and movie metadata (like year, genres, title) to generate the best model

Slide 42

Slide 42 text

Other Approaches Collaborative Filtering “If you and I like the same thing, I might like other things you like” ...Or something like that Other tools can do this pretty easily with userId and movieId See http://spark.apache.org/docs/latest/mllib-collaborative-filtering.html

Slide 43

Slide 43 text

Final Thoughts Garbage In == Garbage Out Well, Garbage Applied == Garbage Out too Can’t just throw algorithms against the wall and see what sticks That’s not data science - your output and user experience will suffer

Slide 44

Slide 44 text

Final Thoughts Compare output from each service to see how they performed Didn’t even talk about cost or pricing so YMMV Consider other approaches Not just about finding well-performing algorithms Must also consider what makes sense for your use case and/or application

Slide 45

Slide 45 text

Final Thoughts These tools are very powerful but they aren’t a panacea Expertise and an understanding of the underlying analyses is critical to making this useful Be careful going down this road - make sure you’ve understood the data science problem before leveraging

Slide 46

Slide 46 text

Thanks! Where can you find me? @crcsmnky http://github.com/crcsmnky sandeep@clusterbeep.org Questions?