Designing better content solutions for personalized recommendations

7563c7a310f9d69458e282fbf0d04bb8?s=47 Ryan Bigge
March 24, 2018

Designing better content solutions for personalized recommendations

These are my slides from IA Summit 2018, where I gave a talk about why building human-friendly sentences from the raw material of machine learning is finicky work.

Shopify’s approach to the problem was to treat personalized recommendations as a design problem. We started by thinking carefully about the mental model(s) users bring to our product. Then we created principles to shape and constrain our recommendations.

By using content and design principles to guide our work, we were able to make our UX as smart as our AI.


Ryan Bigge

March 24, 2018


  1. 1.

    IA Summit — March 24, 2018 Designing better content solutions

    for personalized recommendations Ryan Bigge
  2. 2.

    In Dec 2017, I updated the Chrome app on my

    iPhone. Suddenly, a bunch of suggested articles appeared.
  3. 4.

    Google has trained me to expect a clean white page.

    Suggested articles are a significant product and UX shift.
  4. 6.

    “Interesting articles” based on your Web activity and “unread items

    from your reading list.” That’s a bit thin for such a big product shift.
  5. 7.

    I decided to remove suggested articles. But to do so,

    I had to turn off “Search + Site Suggestions” in my privacy settings.
  6. 8.

    This meant hiding suggested content would also disable predictive search.

    ☹ I did it anyway. That’s how much I disliked those articles.
  7. 9.
  8. 10.

    On Jan 24, 2018, I updated my Chrome app yet

    again. Article suggestions now had an on/off switch
  9. 11.

    Did I just waste 11 slides on a first world

    problem? Nah. Everything I’ll talk about today is in that story.
  10. 12.

    • Personalized recos are a content + design problem •

    Transparency builds (or erodes) user trust • The delicate art of diverting user attention What I’ll talk about today:
  11. 15.

    About a year ago, Jeff and I started talking about

    recommendations. He shared many great ideas and sharpened my thinking. Thanks Jeff.
  12. 17.

    I didn’t grok it all the first time through. That’s

    okay — it forced me to learn more about how to engineer recos.
  13. 18.

    A few months later, my co-worker Putra told me about

    some great academic research about recommender systems. I had no idea this work existed, probably because most of it is behind paywalls.
  14. 19.
  15. 20.

    • “Most online Recommender Systems act like black boxes, not

    offering the user any insight into the system logic or justification for the recommendations.” • “Users perceived that natural language explanations are more trustworthy, contain a more appropriate amount of information, and offer a better user experience.” RecSys insights:
  16. 21.

    • “Designers and data scientists must immerse themselves in the

    other’s approaches to build a common rhythm.” • “It’s hard to gather useful data about users without engaging them, and it’s hard to engage them without having some data to enhance their experience.” More RecSys insights:
  17. 22.

    Christopher Noessel insights: “Trust is built slowly over many interactions,

    and it can all fall quickly with a few failures.” “The practice of design adapts to the thing being designed.” “The majority of users of software cannot open it up to investigate what’s inside, if they suspect some problem.”
  18. 24.

    Whether you’re a newbie or an expert, this isn’t the

    right level of discourse about algorithms.
  19. 26.

    While I love that Arthur C. Clarke quote (“Any sufficiently

    advanced technology is indistinguishable from magic”) Google sauce isn’t just black boxing technology. It’s taking something serious and sophisticated and saying, “Don’t worry your pretty little head about it.”
  20. 27.

    “Yelp’s identity is anchored by a little hamster in a

    rocket ship. This hamster has a name. It’s Hammy.” - Jesse Barron on the “magic sauce” problem in tech
  21. 28.

    “We’re in the middle of a decade of post-dignity design,

    whose dogma is cuteness.” - Jesse Barron on the “magic sauce” problem in tech
  22. 29.

    92% taste match Spotify also obscures the algorithms involved, but

    this is a far more interesting expression of transparency.
  23. 30.

    “We have a great built-in way to only suggest content

    we think you’ll love.” “The more you use Netflix, the more relevant your suggested content will be.”
  24. 31.

    We use a recommendation algorithm that considers: • Genres of

    TV shows + movies • Streaming history and previous ratings • Combined ratings of all Netflix members with similar tastes
  25. 32.

    The tech industry has only recently started to debate the

    ethics + blind spots of algorithms.
  26. 33.
  27. 34.
  28. 35.
  29. 36.

    The language to describe privacy is improving. Hopefully, personalized recos

    will be the next to benefit from increased transparency.
  30. 39.

    Transparency is important for many reasons, including information asymmetry. At

    the risk of oversimplifying our central challenge: we know something the user doesn’t. So we need to share justifications and rationale in a transparent and trustworthy manner.
  31. 41.
  32. 45.

    It looks like you’re writing a letter. Many personalized recommendations

    are: jobs-you-should-do-instead jobs-you-might-want-to-consider
  33. 48.

    • April of 2016 — Shopify acquires Kit CRM •

    Kit is a virtual employee that works with merchants and delivers personalized recos that match their business • Kit uses conversational commerce to deliver those recos Hello from Kit
  34. 49.
  35. 50.

    • Saying no to a reco is almost as valuable

    as saying yes • Designing for feedback is critical • Built around user success, not product retention A bit more about Kit
  36. 52.

    “Design principles are a list of strongly-held opinions that an

    entire team agrees on. They force clarity and reduce ambiguity.” - Emmet Connolly, Intercom
  37. 53.
  38. 55.

    • Relevant — speak to user needs + be as

    specific as possible • Transparent — share the intent behind recommendation • Actionable — make next action clear; clarity over persuasion • Prudent — maintain user trust with high-confidence recos • Cohesive — share the results of taking a recommendation Our 1st draft principles:
  39. 56.

    Our principles have since evolved — we lost prudent, for

    example. And they’ll evolve again. Figuring out how recommendations work across the entire Shopify ecosystem is our next big challenge.
  40. 58.

    Hello, {firstname}. It looks like you want to learn how

    Shopify combined {content strategy} and {design} to improve the {UX} of their personalized marketing recommendations.
  41. 59.

    My talk description pokes fun at Madlib sentences. Better known

    content formulas, they’re still the most common way to scale recos.
  42. 60.
  43. 61.

    Airbnb recos use a content formula, but it works because

    it’s just plugging numbers into a sentence. Content formulas are convenient, but they aren’t always the best way to display a personalized recommendation.
  44. 62.
  45. 64.
  46. 65.

    Boost your chance of getting long-term bookings by 28% Want

    more longer-term bookings? Adding a 17% weekly discount for stays of 7 nights can boost your chance of getting these bookings. Add 17% discount No, thanks
  47. 66.

    Improve your long-term booking rate with a small discount Add

    17% discount No, thanks Recommendation details Predicted outcome: 3 bookings over next 6 weeks Discount required: 17% Discount target: 7 night stays Outcome likelihood: 28%
  48. 67.

    • Develop content + experience principles for recos • Think

    beyond the sentence • Explore content + design implications of transparency Case study summary:
  49. 69.
  50. 70.

    I gave Google a tough time today, but as I

    was putting this talk together, I found an article by Josh Lovejoy about the UX of AI. He lists design principles. He wants Google Clips to solve a real human need. We need more articles like this. It can be a struggle to create good user experiences in far less complicated circumstances. The only way to solve UX and machine learning challenges is to talk more often about our failures and successes.
  51. 71.

    Thanks! We have many worthwhile problems to solve at Shopify.

    Interested? I “recommend” you speak with me! Ryan Bigge