Data Driven Product Development

This talk was delivered during Meta Refresh 2016.

Transcript

1. DATA DRIVEN DEVELOPMENT Product
2. PRODUCT DEVELOPMENT in general
3. HUNCH BASED
4. FOUNDER FORCED
5. REACTIVE
6. CATCH ALL SYSTEMS
7. LACK OF BANDWIDTH
8. DATA DRIVEN DEVELOPMENT Product
9. SHASHANK MEHTA Full time À la carte manager at Razorpay Part time Software Developer Tech Evangelist Marketer Content Writer Salesman @leostatic (slides are a pinned tweet)
10. #MehtaRefresh
11. STEP 1 DEFINE DATA POINTS
12. STEP 0 ASK QUESTIONS the right ones
13. - How many people will use this feature?
 - Does this feature work as expected?
 - What information would you need to debug the feature if it doesn’t work as expected? QUESTIONS
14. - Have we even defined what would be classified as success for this feature?
 - How much of a success is it really? Can it be quantified?
 - What would make it a failed feature? QUESTIONS
16. STEP 1 DEFINE DATA POINTS
17. STEP 2 COLLECT DATA
18. STEP 3 ANALYSE THE DATA
19. STEP 4 ITERATE, TRACK, ITERATE
20. STEP 5 PROFIT!
21. TOO ABSTRACT?
22. INSTAGRAM STORIES Credits: blog.instagram.com
23. INSTAGRAM STORIES Credits: blog.instagram.com Post Photos or Videos through the + button
 Keep posting throughout the day Stories from your friends are on the top
 Click on a friend’s photo to see their story
24. INSTAGRAM STORIES Credits: blog.instagram.com Add filters like always Draw or Write on the shots The white bar on the top indicates the count Click on the bottom bar to send a message Tap left or right to go to previous or next shot in this story
 
 Swipe left or right to skip a friend

25. STEP 0 ASK QUESTIONS the right ones
26. - How many people will use this feature?
 - Does this feature work as expected?
 - What information would you need to debug the feature if it doesn’t work as expected? QUESTIONS
27. - Have we even defined what would be classified as success for this feature?
 - How much of a success is it really? Can it be quantified?
 - What would make it a failed feature? QUESTIONS
28. - Number and type of Users
 - Quantifying the usage or effect DEFINING SUCCESS / FAILURE
29. DEFINING SUCCESS Credits: blog.instagram.com SUCCESS: On a day, 85% of active users post at least 3 shots and view at least 4 stories FAILURE: On a day, less than 50% of active users post less than 3 shots and view less than 3 stories
30. STEP 1 DEFINE DATA POINTS
31. DATA POINTS Credits: blog.instagram.com For deciding between Success & Failure, track per user - Number of stories viewed - No of posts in the user’s story That’s it! Really?
32. ANALYTICS A PM SHOULD WORRY ABOUT and the developer/designer too
33. ANALYTICS A PM SHOULD WORRY ABOUT Database vs Ephemeral
34. EPHEMERAL DATA POINTS Credits: blog.instagram.com - Time taken to post a photo in a story - From app opening to posting - Mark out delays due to network and interface
 - Days before posting first ever story
 - No of stories viewed before posting first ever story

35. Credits: blog.instagram.com - Use of various gestures - No of times each gesture is used - Total Stories viewed before using a gesture
 - Use of features like drawing or writing
 - No of messages sent from Stories EPHEMERAL DATA POINTS
36. STEP 2 COLLECT DATA
37. TRACKING DATA
38. STEP 3 ANALYSE THE DATA
39. ANALYSING DATA Credits: blog.instagram.com Let’s analyse Gestures
 - Is a gesture useful? - No? - Should we remove it or improve it? Product call
 - Is a gesture intuitive? - No? - Can a more intuitive gesture be used? - Is the gesture important enough to be called out? - Can it be explained in a non-intrusive way?
40. STEP 4 ITERATE, TRACK, ITERATE
41. STEP 5 PROFIT!
42. MOAR EXAMPLES!
43. MIXPANEL FTW! Success Rates
44. MIXPANEL FTW!
45. MIXPANEL FTW!
46. AUTO-OTP FILLING
47. DEFINING SUCCESS / FAILURE - Number and type of Users 
 - Quantifying the usage or effect
48. DEFINING SUCCESS / FAILURE - Number and type of Users everyone on android
 - Quantifying the usage or effect
49. - Number and type of Users everyone
 - Quantifying the usage or effect success rate and time to pay DEFINING SUCCESS / FAILURE
50. SUCCESS: On a weekly average, reduce time taken to complete a payment by 30% and increase success rates by 5% FAILURE: Any decrease in success rates or increase in time taken to complete a payment, even by milliseconds AUTO-OTP FILLING
51. RESULTS Increase in Success Rates 8% Decrease in time to pay 45%

52. ITERATIONS Overlay was added to make the bottom part more prominent 
 But what if OTP wasn’t received?
 
 Expected User will click through to access bank page
 Reality ~10% did not realise it’s clickable
53. ITERATIONS
54. NEW POSSIBILITIES
55. INSTAGRAM EXPERIMENTS Credits: blog.instagram.com Once the base is set, whole lot of experiments can then be run. For eg: - Does the order of friends in the stories list affect any stat?
 - How much of an effect do notifications have?
 - When clicking a photo via the app, are prompts to add the photo to Story useful?
56. FOR GOOD KARMA Background images are from unsplash.com Creative Commons 0 licensed Thanks for all the great work!
57. Credits: @pablostanley https://thedesignteam.io/from-a-product-perspective-2f5185a43827 QUESTIONS! @leostatic #MehtaRefresh