Leveraging machine learning to optimize manual review processes

This talk is about how Sittercity plans to scale our existing on-demand babysitting business to a national stage, whilst maintaining quality, safety and mitigating risk within the marketplace.

We look at some of the challenges we face as we bring new dynamics to parent/sitter relationships to a national scale; we examine one specific process that costs us and our sitters a lot of time/money; and understand how developments in machine learning are opening up a lot of opportunity in this area, without compromising on quality or adding risk.

Transcript

1. Leveraging Machine Learning to optimize manual review processes By Sam de Freyssinet 2018, Creative Commons
2. 3.2M families 5.5M sitters 120K monthly new users over 2500 join daily connect a family with care giver every 9 seconds
3. Instant booking and payment Improved messaging Favorites management and booking Accurate sitter availability 98% Customer satisfaction
4. Our mission To make childcare finally work. Our strategy We are building a smarter matching experience and 
 scaling instant booking nationally for busy families 
 and sitters who love this work.
5. Sunflower, Copyright © 2016 Sam de Freyssinet
6. opencv.org Open souce library Bring your own training data Object detection C++ library with bindings for most languages
7. Phone cameras are not good Processing on a mobile takes a long time and lots of energy Training data not available Computer Vision is hard in 2010
8. tensorflow.org/tutorials/image_retraining
9. since 2001
10. Defining classification of bad actors for many years since 2001
11. Risk in our marketplace Jobs Messaging Purchases User to user payment People
12. Jobs Messaging Purchases User to user payment People Risk in our marketplace
13. Sittercity babysitter profile pictures from the Chicago area, Spring 2018
14. Crystal L babysitter profile, Copyright © 2018 Sittercity - sittercity.com/babysitters/il/chicago/13025116
15. Profile Review Profile Review Create Profile Profile Review Profile Publish Pass Review? Edit Profile No Yes Sitter on-boarding
16. Profile Review Profile Review Create Profile Profile Review Profile Publish Pass Review? Edit Profile No Yes Sitter on-boarding 1 hour 1 sec 1 day 1 sec ≈ 25 hours 2 sec
18. Sittercity babysitter profile pictures from the Chicago area, Spring 2018
19. Sittercity babysitter profile pictures from the Chicago area, Spring 2018 headshot must clearly show from your chin to forehead the photo should be just of you
20. Learning as a Service Machine Learning Engine + Vision API Microsoft Azure Cognitive Services Computer Vision Amazon Web Services Rekognition
21. Learning as a Service Machine Learning Engine + Vision API Microsoft Azure Cognitive Services Computer Vision Amazon Web Services Rekognition Other providers
26. Face detection and multiple people Nudity and adult Object labels Inappropriate text Annotations and filters Clothing 97% 96% 97% 95% 92% 90% struggled with multiple people in early testing 99% 99% 96% 94% 94% 95% 5% <1% <1% Results compiled from sample of 1000 profile pictures from the internet, tested in March 2018. Numbers represent percentage of positive identification within the eightieth percentile. 96% 93% 96% snapchat detection using web indexes
27. Snapchat filters Example images from Soar Astronova and Sophie Thomas. Images contain Snapchat Filters, Copyright © Snapchat 2017
28. Machine Learning Engine + Vision API Sittercity has recently changed tooling on Google Cloud Platform to use new introduced ML services
29. Sunglasses Headwear Snapchat filters Trouble recognizing sunglasses in place of eyeglasses Hats or headwear obscuring face hard to detect Snapchat filters not identifiable with any certainty
30. Selected Solution Machine Learning Engine + Vision API TensorFlow OpenSource Machine Learning Framework and API Sittercity has recently changed tooling on Google Cloud Platform to use new introduced ML services
31. tensorflow.org
32. Training the new model 750,000 rejected profile pictures and counting Rejected Sittercity babysitter profile pictures from the Chicago area, Spring 2018
33. No classification
34. DIY classification Save Image 8,234 of 767,034 Multiple People Not Clear Photo Too Far Away Offensive Headwear Unprofessional Snapchat Filters Sunglasses Sunglasses
35. DIY classification Sunglasses 1 week to classify 1,000 images Save Image 8,187 of 767,034 Multiple People Not Clear Photo Too Far Away Offensive Headwear Unprofessional Snapchat Filters Sunglasses 82% label accuracy
36. Training datasets Training set Training set is used to train the model on the problem Rejected Sittercity babysitter profile pictures from the Chicago area, Spring 2018
37. Cleaning datasets Producing a good training set Cleansed training data is vital to avoid overfitting the model Rejected Sittercity babysitter profile pictures from the Chicago area, Spring 2018
38. Cleaning datasets Producing a good training set Cleansed training data is vital to avoid overfitting the model Overfitting will cause the model to learn unwanted fluctuations or random noise Rejected Sittercity babysitter profile pictures from the Chicago area, Spring 2018
39. image-net.org kaggle.com
40. image-net.org/synset
41. Transfer learning Standing on the shoulders of giants Use existing pre-trained image classifiers built upon millions of images Layer your domain specific classifier on top of a preexisting image classifier model Reduces training effort Model A Model B Pre-trained classifier New classification
42. Verifying trained models Training set Training set is used to train the model on the problem Test set Test set is used to verify the model is performing within expected parameters ≠
43. Eyewear Sunglasses Glasses Vision Care Cool Vacation Selfie 98% 97% 95% 92% 82% 80% 76% TensorFlow image classification wall time, 0.2 seconds using Nvidia Titan Xp GPU and Intel i7 8700k 32GB RAM IMG_0476.jpg
44. Save Image 8,234 of 767,034 Multiple People Not Clear Photo Too Far Away Offensive Headwear Unprofessional Snapchat Filters Sunglasses Updated review process Rejected pictures are classified Building a new verified classification dataset for future classification problems
45. Profile Review Profile Review Create Profile Review Publish Pass? Edit Profile Yes Revised sitter on-boarding 1 hour 1 sec (1 min) 1 sec ≈ 1 hour Manual Review No
46. Improve UX Take photo ! We have found some issues: • There appears to be two people in the picture. • One of the subjects is wearing sunglasses. More information TensorFlow Mobile Execute TensorFlow trained models directly on modern phone hardware
47. Sunglasses Headwear Snapchat filters Now supported and accurate detection achieved We are custom training a obscured face model Working with Google to custom train SnapChat filter model
48. Something new AutoML
49. Thank you! Questions? I will be hanging around after the presentation to answer any other questions. Interested in solving problems like this? Lets chat! Please send your résumé to sdefreyssinet@sittercity.com is hiring
50. This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License. Some rights reserved. SITTERCITY AND CHIME APPS …………….………………………… PHOTOS Sittercity BBC MICRO COMPUTER ….……………………….………………… Wikipedia WESTWORLD, SEASON 2 …………….….……….….……… Home Box Office THE MATRIX …………………………….…………….…………….. Warner Bros SUNFLOWER IMAGE ………………………………….….…. Sam de Freyssinet SITTERCITY CHILD .….…………….…….….….……….……………… Sittercity SITTERCITY SITTERS ……………….…….….….…….….….………… Sittercity SNAPCHAT FILTER MODELS .………….….….….…..…….…. Soar Astronova Sophie Thomas MARKER PEN CHECKLIST PHOTO ..…………….…….…..….……..…. Pixbay WRITTEN AND PRESENTED BY Sam de Freyssinet SLIDES Sam de Freyssinet Slides produced using Keynote Additional material produced using “Sittercity” and the “Sittercity logo “ are registered trademarks of Sittercity, Inc. All rights reserved. “Westworld”, “HBO”, and the “HBO logo” are registered trademarks of Home Box Office, Inc a Time Warner Company. All rights reserved. “The Matrix”, “Warner Bros”, the “Warner Bros logo” are registered trademarks of Warner Bros. Entertainment, Inc a Time Warner Company. All rights reserved. “OpenCV” and the “OpenCV logo” are registered trademarks of the OpenCV team. All rights reserved. “TensorFlow” and the “TensorFlow logo” are registered trademarks of Google, Inc an Alphabet Company. All rights reserved. “Google”, the “Google logo”, “Google Cloud”, the “Google Cloud logo”, “Google Cloud Vision”, the “Google Cloud Vision logo”, “Google Cloud Machine Learning”, the “Google Cloud Machine Learning logo”, “Google Cloud AutoML” and the “Google Cloud AutoML logo” are registered trademarks of Google, Inc an Alphabet Company. All rights reserved. “Microsoft”, “Microsoft Azure”, the “Microsoft Azure Logo”, “Microsoft Azure Cognitive Services Computer Vision” and “the Microsoft Azure Cognitive Services Computer Vision logo” are registered trademarks of the Microsoft Corporation. All rights reserved. “Amazon Web Services”, the “Amazon Web Services logo”, “Amazon Web Services Rekognition” and the “Amazon Web Services Rekognition Logo” are registered trademarks of Amazon.com Inc. All rights reserved. © MMXVIII Sketch Reflector Final Cut Pro