Machine Learning from a Product Perspective

MACHINE LEARNING FROM A PRODUCT PERSPECTIVE

The trends of big data and cost-effective computing power have given way to an unprecedented focus on developing machine learning algorithms and open source tools, making machine learning more accessible and powerful today than ever. But building machine learning products is about much more than picking the right algorithm or library. It's a product manager's job to use machine learning for the right problems, and to make sure the end user experience solves those problems and fosters user trust. This talk will illustrate machine learning concepts through applications in products, provide a framework for determining problems that are good for machine learning to solve, and talk about how the product development cycle for machine learning-enabled products might differ from traditional software.

Agenda:
- What is ML?
- What problems are good for ML?
- What should product managers bring to the table?
- ML product development cycle
- Why machine learning now?

Transcript

1. MACHINE LEARNING FROM A PRODUCT PERSPECTIVE I N G A C H E N
2. @ I N G A C H E N S I L I C O N VA L L E Y N AT I V E D A U G H T E R O F S O F T WA R E + H A R D WA R E E N G I N E E R S I N V E S T M E N T B A N K E R T E C H M & A J E F F E R I E S S I L I C O N VA L L E Y E A R LY B U S I N E S S H I R E I o T S TA R T U P F O R C A R S A U T O M AT I C L A B S S A N F R A N C I S C O P R O D U C T M A N A G E R D ATA I N S I G H T S & F L E E T A U T O M AT I C L A B S S A N F R A N C I S C O P R O D U C T M A N A G E R A N A LY T I C S S Q U A R E S PA C E N E W YO R K C I T Y
3. Millions of miles of driving Millions of websites Slice of the internet Drivers Fuel efficiency Car Maintenance SMBs Online presence and business
4. Millions of miles of driving Millions of websites Slice of the internet Create an optimized experience for each user MACHINE LEARNING
5. B E YO N D T H E A L G O R I T H M : B U I L D I N G M L P R O D U C T S W H AT S H O U L D P R O D U C T P E O P L E B R I N G T O T H E TA B L E ? B A C KG R O U N D : W H AT I S M L? W H AT P R O B L E M S A R E G O O D F O R M L? W H Y N O W MACHINE LEARNING FROM A PRODUCT PERSPECTIVE
6. B E YO N D T H E A L G O R I T H M : B U I L D I N G M L P R O D U C T S W H AT P R O B L E M S A R E G O O D F O R M L? W H Y N O W MACHINE LEARNING FROM A PRODUCT PERSPECTIVE B A C KG R O U N D : W H AT I S M L? W H AT S H O U L D P R O D U C T P E O P L E B R I N G T O T H E TA B L E ?
7. Artificial Intelligence Human Intelligence Exhibited by Machines
8. GOALS OF AI: Reasoning Knowledge Planning Learning Natural Language Processing Perception Move & manipulate objects Artificial Intelligence Human Intelligence Exhibited by Machines
9. “[Machine Learning is the] field of study that gives computers the ability to learn without being explicitly programmed Arthur Samuel, 1959
10. “[Machine Learning is the] field of study that gives computers the ability to learn without being explicitly programmed (often more data than is humanly possible to go through) and makes a determination or prediction about something in the world. by observing data
11. H O W D O E S M A C H I N E L E A R N I N G W O R K ?
12. INPUT OUTPUT ML MODEL EMAIL SPAM NOT SPAM
13. DATA EMAILS
14. “CORRECT ANSWERS” IDEALLY…IT’S LABELED EMAILS LABELED AS SPAM / NOT SPAM
15. 70% Training Set Validation Set 30% TEACHER GIVES EXAMPLES TO ILLUSTRATE A CONCEPT TEST
16. B E YO N D T H E A L G O R I T H M : B U I L D I N G M L P R O D U C T S B A C KG R O U N D : W H AT I S M L? W H AT P R O B L E M S A R E G O O D F O R M L? W H Y N O W MACHINE LEARNING FROM A PRODUCT PERSPECTIVE W H AT S H O U L D P R O D U C T P E O P L E B R I N G T O T H E TA B L E ?
17. D E C I S I O N S W I T H M A N Y I N P U T S and it’s unclear how to combine them W H AT P R O B L E M S A R E G O O D F O R M A C H I N E L E A R N I N G ?
18. W H O W I L L W I N T H E E L E C T I O N ? Hillary vs Trump Education Income level Race Gender Geographic location Religion What car you drive
21. D E C I S I O N S W I T H M A N Y O U T P U T S and it’s infeasible to show all outputs W H AT P R O B L E M S A R E G O O D F O R M A C H I N E L E A R N I N G ?
22. S L A C K H I G H L I G H T S M A N A G E T H E D E L U G E O F W O R K M E S S A G E S
23. M A K I N G D E C I S I O N S AT S C A L E for millions/billions of items or users W H AT P R O B L E M S A R E G O O D F O R M A C H I N E L E A R N I N G ?
24. ML MODEL OUTPUT PREDICTED LABELS CONFIDENCE % CAT CAT NOT NOT NOT SUPERVISED LEARNING CAT NOT CAT CAT NOT TRAINING SET UNSEEN DATA INPUT VALIDATION SET I M A G E S E A R C H “ C AT ”
25. D E C I S I O N S W I T H M A N Y I N P U T S D E C I S I O N S W I T H M A N Y O U T P U T S M A K I N G D E C I S I O N S AT S C A L E W H AT P R O B L E M S A R E G O O D F O R M A C H I N E L E A R N I N G ?
26. PREDICT DISCOVER OPTIMIZE DETECT RANK AUTOMATE stock prices housing prices sales health outcomes pricing user experience (personalization) recommendations bank fraud fake news fractures road hazards search results newsfeed recommendations patterns trends customer segmentation human tasks not easily delineated by rules W H AT P R O B L E M S A R E G O O D F O R M A C H I N E L E A R N I N G ?
27. U N S U P E R V I S E D L E A R N I N G D I S C O V E R PAT T E R N S
28. P R O B L E M How should I improve my fuel efficiency based on my driving behavior?
30. 8 clusters
31. What distinguishes those 8 clusters?
32. What are the biggest factors affecting fuel efficiency of our drivers? Aggression on highway Impatience on local roads Weather, terrain, traffic
33. NY Cabbie Grocery Go-Getter Highway Hustler Racecar Driver Suburban Scout Foxy Fighter Pilot Peaceful Sage Commuter
35. A U T O M AT E C O M P L E X TA S K S D E E P L E A R N I N G
37. S E L F - D R I V I N G C A R S
38. P R O B L E M A slice of the internet: How do we build products for such a diverse user base?
43. S O L U T I O N VISUAL SEARCH ENGINE FOR WEBSITES P R O B L E M How do we build products for such a diverse user base?
45. Deep Learning: Under the hood (SIMPLIFIED) Convolutional variational auto-encoder Recreate the image Find similarities Learns how to interpret the structure of an image hidden layers DEEP
48. B E YO N D T H E A L G O R I T H M : B U I L D I N G M L P R O D U C T S B A C KG R O U N D : W H AT I S M L? W H AT P R O B L E M S A R E G O O D F O R M L? W H Y N O W MACHINE LEARNING FROM A PRODUCT PERSPECTIVE W H AT S H O U L D P R O D U C T P E O P L E B R I N G T O T H E TA B L E ?
49. 1. IDENTIFY THE RIGHT PROBLEMS TO USE ML TO SOLVE
50. 2. FIND OUT: DO YOU HAVE THE DATA AND IS IT GOOD?
51. 3. CONSIDER THE NON-ML ALTERNATIVE Is that good enough?
52. 4. KNOW HOW INPUTS + OUTPUTS ARE RELATED
53. 5. DESIGN FOR WHEN THE MODEL FAILS Because it will
54. USER TRUST Easy to lose Hard to win back
55. W H AT S H O U L D P R O D U C T P E O P L E B R I N G T O T H E TA B L E ? B A C KG R O U N D : W H AT I S M L? W H AT P R O B L E M S A R E G O O D F O R M L? W H Y N O W MACHINE LEARNING FROM A PRODUCT PERSPECTIVE B E YO N D T H E A L G O R I T H M : B U I L D I N G M L P R O D U C T S
56. PRIORITIZE BUILD LAUNCH LEARN MEASURE FOR SOFTWARE PRODUCTS P R O D U C T D E V E L O P M E N T C YC L E
57. CLASSIFY AS ML CONSIDER NON-ML GATHER & PREP DATA PRODUCT CHANGES? BUILD & EVALUATE MODELS THROW AWAY IDENTIFY MODEL WEAKNESSES COLLECT MORE DATA REBUILD & IMPROVE LAUNCH PRODUCTION MODEL ACCURACY % TRADEOFFS: ACCURACY VS COMPUTE POWER VS VISIBILITY FOR SOFTWARE PRODUCTS USING ML P R O D U C T D E V E L O P M E N T C YC L E PRIORITIZE BUILD LAUNCH LEARN MEASURE
58. INPUT OUTPUT MANY ML MODELS P R O D U C T S I N P R A C T I C E
59. B A C KG R O U N D : W H AT I S M L? B E YO N D T H E A L G O R I T H M : B U I L D I N G M L P R O D U C T S W H AT S H O U L D P R O D U C T P E O P L E B R I N G T O T H E TA B L E ? W H AT P R O B L E M S A R E G O O D F O R M L? W H Y N O W MACHINE LEARNING FROM A PRODUCT PERSPECTIVE L E A R N I N G F R O M D ATA W I T H O U T B E I N G E X P L I C I T LY P R O G R A M M E D M A N Y I N P U T S , M A N Y O U T P U T S M A K I N G D E C I S I O N S AT S C A L E D E S I G N F O R W H E N T H E M O D E L FA I L S L A U N C H , R E B U I L D & I M P R O V E . T R A D E O F F S
60. B E YO N D T H E A L G O R I T H M : B U I L D I N G M L P R O D U C T S W H AT YO U N E E D F O R M L B A C KG R O U N D : W H AT I S M L? W H AT P R O B L E M S A R E G O O D F O R M L? MACHINE LEARNING FROM A PRODUCT PERSPECTIVE W H Y N O W
61. MACHINE LEARNING PRODUCTS MATURING OF ML ALGORITHMS with open source tools and deep learning MASSIVE COMPUTE POWER Cloud computation GPU processing ACCESSIBILITY OF OPEN DATA CommonCrawl - petabytes ImageNet - 14M labeled C O N V E R G I N G T R E N D S … YOUR COMPETITORS ARE DOING IT NOT JUST FOR BIG COMPANIES EVEN INDIVIDUALS CAN DO IT
62. BUILD PRODUCTS THAT KNOW YOU BETTER THAN YOU KNOW YOURSELF CONSUMER M A C H I N E L E A R N I N G E N A B L E S …
63. YOUR DRIVING
64. “I wish my significant other understood me as well as Discover Weekly.” YOUR MUSIC TASTES
65. EVERYTHING YOU BUY
66. BUILD PRODUCTS THAT HELP BUSINESSES BE SUCCESSFUL SMB/ENTERPRISE M A C H I N E L E A R N I N G E N A B L E S …
67. YOUR WORKFLOW
68. EVERYTHING WE KNOW ABOUT SUCCESSFUL WEBSITES EVERYTHING WE KNOW ABOUT YOUR BUSINESS GOALS BUILD A SUCCESSFUL BUSINESS ONLINE
69. twitter: @ingachen THANK YOU!