Designing better content solutions for personalized recommendations

These are my slides from IA Summit 2018, where I gave a talk about why building human-friendly sentences from the raw material of machine learning is finicky work.

Shopify’s approach to the problem was to treat personalized recommendations as a design problem. We started by thinking carefully about the mental model(s) users bring to our product. Then we created principles to shape and constrain our recommendations.

By using content and design principles to guide our work, we were able to make our UX as smart as our AI.

Transcript

1. IA Summit — March 24, 2018
   Designing better content solutions
   for personalized recommendations
   Ryan Bigge
   

   View Slide

2. In Dec 2017, I updated the
   Chrome app on my iPhone.
   Suddenly, a bunch of
   suggested articles appeared.
   

   View Slide

3. The articles annoyed me.
   

   View Slide

4. Google has trained me to expect a clean white page.
   Suggested articles are a significant product and UX shift.
   

   View Slide

5. Where’s the onboarding Google?
   Sell me on the value, Google!
   

   View Slide

6. “Interesting articles” based
   on your Web activity and
   “unread items from your
   reading list.”
   That’s a bit thin for such a
   big product shift.
   

   View Slide

7. I decided to remove
   suggested articles.
   But to do so, I had to
   turn off “Search + Site
   Suggestions” in my
   privacy settings.
   

   View Slide

8. This meant hiding suggested
   content would also disable
   predictive search. ☹
   I did it anyway. That’s how
   much I disliked those articles.
   

   View Slide

9. View Slide

10. On Jan 24, 2018, I
    updated my Chrome
    app yet again.
    Article suggestions
    now had an on/off
    switch
    

    View Slide

11. Did I just waste 11
    slides on a first
    world problem?
    Nah.
    Everything I’ll talk
    about today is in
    that story.
    
    

    View Slide

12. • Personalized recos are a content + design problem
    • Transparency builds (or erodes) user trust
    • The delicate art of diverting user attention
    What I’ll talk about today:
    

    View Slide

13. Learning about
    machine learning
    
    

    View Slide

14. This is my friend Jeff MacIntyre
    

    View Slide

15. About a year ago, Jeff and I started talking about
    recommendations. He shared many great ideas
    and sharpened my thinking.
    Thanks Jeff.
    

    View Slide

16. The article that got me started:
    

    View Slide

17. I didn’t grok it all the
    first time through.
    That’s okay — it
    forced me to learn
    more about how to
    engineer recos.
    

    View Slide

18. A few months later, my co-worker Putra told me
    about some great academic research about
    recommender systems.
    I had no idea this work existed, probably because
    most of it is behind paywalls.
    

    View Slide

19. View Slide

20. • “Most online Recommender Systems act like black
    boxes, not offering the user any insight into the system
    logic or justification for the recommendations.”
    • “Users perceived that natural language explanations are
    more trustworthy, contain a more appropriate amount of
    information, and offer a better user experience.”
    RecSys insights:
    

    View Slide

21. • “Designers and data scientists must immerse themselves
    in the other’s approaches to build a common rhythm.”
    • “It’s hard to gather useful data about users without
    engaging them, and it’s hard to engage them without
    having some data to enhance their experience.”
    More RecSys insights:
    

    View Slide

22. Christopher Noessel insights:
    “Trust is built slowly over many
    interactions, and it can all fall quickly
    with a few failures.”
    “The practice of design adapts to the
    thing being designed.”
    “The majority of users of software
    cannot open it up to investigate what’s
    inside, if they suspect some problem.”
    

    View Slide

23. Transparency
    
    Transparency
    Transparency
    

    View Slide

24. Whether you’re a newbie or an expert, this isn’t
    the right level of discourse about algorithms.
    

    View Slide

25. Important according to Google magic
    Important according to our magic sauce
    
    

    View Slide

26. While I love that Arthur C. Clarke quote (“Any sufficiently
    advanced technology is indistinguishable from magic”)
    Google sauce isn’t just black boxing technology.
    It’s taking something serious and sophisticated and
    saying, “Don’t worry your pretty little head about it.”
    

    View Slide

27. “Yelp’s identity is anchored
    by a little hamster in a rocket
    ship. This hamster has a
    name. It’s Hammy.”
    - Jesse Barron on the “magic sauce” problem in tech
    

    View Slide

28. “We’re in the middle of a
    decade of post-dignity design,
    whose dogma is cuteness.”
    - Jesse Barron on the “magic sauce” problem in tech
    

    View Slide

29. 92% taste match
    Spotify also obscures the algorithms involved, but this is a
    far more interesting expression of transparency.
    

    View Slide

30. “We have a great built-in
    way to only suggest content
    we think you’ll love.”
    “The more you use Netflix,
    the more relevant your
    suggested content will be.”
    

    View Slide

31. We use a recommendation
    algorithm that considers:
    • Genres of TV shows + movies
    • Streaming history and
    previous ratings
    • Combined ratings of all Netflix
    members with similar tastes
    

    View Slide

32. The tech industry has only recently started to
    debate the ethics + blind spots of algorithms.
    

    View Slide

33. View Slide

34. View Slide

35. View Slide

36. The language to describe privacy is improving.
    Hopefully, personalized recos will be the next to
    benefit from increased transparency.
    

    View Slide

37. “Privacy is a fundamental human right.”
    

    View Slide

38. Not sure if this is related to algorithms or
    privacy, but it’s a good sign.
    

    View Slide

39. Transparency is important for many reasons,
    including information asymmetry.
    At the risk of oversimplifying our central challenge:
    we know something the user doesn’t.
    So we need to share justifications and rationale in
    a transparent and trustworthy manner.
    

    View Slide

40. There’s a big power
    imbalance in big data.
    

    View Slide

41. View Slide

42. Fair trade algorithms
    Organic algorithms
    Certified free of dark patterns
    

    View Slide

43. Jobs-to-be-done
    

    View Slide

44. Many personalized
    recommendations are:
    jobs-you-should-do-instead
    jobs-you-might-want-to-consider
    

    View Slide

45. It looks like you’re writing a letter.
    Many personalized
    recommendations are:
    jobs-you-should-do-instead
    jobs-you-might-want-to-consider
    

    View Slide

46. Mini-case study
    ⚗
    

    View Slide

47. A little bit about Kit
    

    View Slide

48. • April of 2016 — Shopify acquires Kit CRM
    • Kit is a virtual employee that works with merchants and
    delivers personalized recos that match their business
    • Kit uses conversational commerce to deliver those recos
    Hello from Kit
    

    View Slide

49. View Slide

50. • Saying no to a reco is almost as valuable as saying yes
    • Designing for feedback is critical
    • Built around user success, not product retention
    A bit more about Kit
    

    View Slide

51. Design principles
    

    View Slide

52. “Design principles are a list of
    strongly-held opinions that an
    entire team agrees on. They force
    clarity and reduce ambiguity.”
    - Emmet Connolly, Intercom
    

    View Slide

53. View Slide

54. • Aspirational
    • Transparent
    • Controllable
    • Challenging
    • Authentic
    Bibblio believes recos should be:
    

    View Slide

55. • Relevant — speak to user needs + be as specific as possible
    • Transparent — share the intent behind recommendation
    • Actionable — make next action clear; clarity over persuasion
    • Prudent — maintain user trust with high-confidence recos
    • Cohesive — share the results of taking a recommendation
    Our 1st draft principles:
    

    View Slide

56. Our principles have since evolved — we lost
    prudent, for example. And they’ll evolve again.
    Figuring out how recommendations work across the
    entire Shopify ecosystem is our next big challenge.
    

    View Slide

57. Beyond the sentence
    

    View Slide

58. Hello, {firstname}. It looks like
    you want to learn how Shopify
    combined {content strategy}
    and {design} to improve the
    {UX} of their personalized
    marketing recommendations.
    

    View Slide

59. My talk description pokes fun at Madlib sentences.
    Better known content formulas, they’re still the
    most common way to scale recos.
    

    View Slide

60. View Slide

61. Airbnb recos use a content formula, but it works
    because it’s just plugging numbers into a sentence.
    Content formulas are convenient, but they aren’t always
    the best way to display a personalized recommendation.
    

    View Slide

62. View Slide

63. Content formulas only
    go so far with detailed
    taxonomies.
    

    View Slide

64. View Slide

65. Boost your chance of getting
    long-term bookings by 28%
    Want more longer-term bookings? Adding a
    17% weekly discount for stays of 7 nights
    can boost your chance of getting these
    bookings.
    Add 17% discount
    No, thanks
    

    View Slide

66. Improve your long-term booking rate
    with a small discount
    Add 17% discount
    No, thanks
    Recommendation details
    Predicted outcome: 3 bookings over next 6 weeks
    Discount required: 17%
    Discount target: 7 night stays
    Outcome likelihood: 28%
    

    View Slide

67. • Develop content + experience principles for recos
    • Think beyond the sentence
    • Explore content + design implications of transparency
    Case study summary:
    

    View Slide

68. Conclusion
    
    

    View Slide

69. View Slide

70. I gave Google a tough time today, but as I was putting this talk
    together, I found an article by Josh Lovejoy about the UX of AI.
    He lists design principles.
    He wants Google Clips to solve a real human need.
    We need more articles like this. It can be a struggle to create
    good user experiences in far less complicated circumstances.
    The only way to solve UX and machine learning challenges is
    to talk more often about our failures and successes.
    

    View Slide

71. Thanks!
    We have many worthwhile problems to solve at Shopify.
    Interested? I “recommend” you speak with me!
    Ryan Bigge
    

    View Slide

72. Appendix
    (
    

    View Slide

73. • https://medium.com/airbnb-engineering/how-we-deliver-insights-to-hosts-7d836520a38
    • http://reallifemag.com/the-babysitters-club/
    • http://nymag.com/vindicated/2016/10/clippy-didnt-just-annoy-you-he-changed-the-world.html
    • https://blog.intercom.com/principles-bot-design
    • http://www.bibblio.org/about
    • https://www.google.com/basepages/producttype/taxonomy.en-US.txt
    • https://design.google/library/ux-ai/
    • https://thisisbucket.com/
    References and resources
    

    View Slide