Designing better content solutions for personalized recommendations

These are my slides from IA Summit 2018, where I gave a talk about why building human-friendly sentences from the raw material of machine learning is finicky work.

Shopify’s approach to the problem was to treat personalized recommendations as a design problem. We started by thinking carefully about the mental model(s) users bring to our product. Then we created principles to shape and constrain our recommendations.

By using content and design principles to guide our work, we were able to make our UX as smart as our AI.

Transcript

1. IA Summit — March 24, 2018 Designing better content solutions for personalized recommendations Ryan Bigge
2. In Dec 2017, I updated the Chrome app on my iPhone. Suddenly, a bunch of suggested articles appeared.
3. The articles annoyed me.
4. Google has trained me to expect a clean white page. Suggested articles are a significant product and UX shift.
5. Where’s the onboarding Google? Sell me on the value, Google!
6. “Interesting articles” based on your Web activity and “unread items from your reading list.” That’s a bit thin for such a big product shift.
7. I decided to remove suggested articles. But to do so, I had to turn off “Search + Site Suggestions” in my privacy settings.
8. This meant hiding suggested content would also disable predictive search. ☹ I did it anyway. That’s how much I disliked those articles.
10. On Jan 24, 2018, I updated my Chrome app yet again. Article suggestions now had an on/off switch
11. Did I just waste 11 slides on a first world problem? Nah. Everything I’ll talk about today is in that story.
12. • Personalized recos are a content + design problem • Transparency builds (or erodes) user trust • The delicate art of diverting user attention What I’ll talk about today:
13. Learning about machine learning
14. This is my friend Jeff MacIntyre
15. About a year ago, Jeff and I started talking about recommendations. He shared many great ideas and sharpened my thinking. Thanks Jeff.
16. The article that got me started:
17. I didn’t grok it all the first time through. That’s okay — it forced me to learn more about how to engineer recos.
18. A few months later, my co-worker Putra told me about some great academic research about recommender systems. I had no idea this work existed, probably because most of it is behind paywalls.
20. • “Most online Recommender Systems act like black boxes, not offering the user any insight into the system logic or justification for the recommendations.” • “Users perceived that natural language explanations are more trustworthy, contain a more appropriate amount of information, and offer a better user experience.” RecSys insights:
21. • “Designers and data scientists must immerse themselves in the other’s approaches to build a common rhythm.” • “It’s hard to gather useful data about users without engaging them, and it’s hard to engage them without having some data to enhance their experience.” More RecSys insights:
22. Christopher Noessel insights: “Trust is built slowly over many interactions, and it can all fall quickly with a few failures.” “The practice of design adapts to the thing being designed.” “The majority of users of software cannot open it up to investigate what’s inside, if they suspect some problem.”
23. Transparency Transparency Transparency
24. Whether you’re a newbie or an expert, this isn’t the right level of discourse about algorithms.
25. Important according to Google magic Important according to our magic sauce
26. While I love that Arthur C. Clarke quote (“Any sufficiently advanced technology is indistinguishable from magic”) Google sauce isn’t just black boxing technology. It’s taking something serious and sophisticated and saying, “Don’t worry your pretty little head about it.”
27. “Yelp’s identity is anchored by a little hamster in a rocket ship. This hamster has a name. It’s Hammy.” - Jesse Barron on the “magic sauce” problem in tech
28. “We’re in the middle of a decade of post-dignity design, whose dogma is cuteness.” - Jesse Barron on the “magic sauce” problem in tech
29. 92% taste match Spotify also obscures the algorithms involved, but this is a far more interesting expression of transparency.
30. “We have a great built-in way to only suggest content we think you’ll love.” “The more you use Netflix, the more relevant your suggested content will be.”
31. We use a recommendation algorithm that considers: • Genres of TV shows + movies • Streaming history and previous ratings • Combined ratings of all Netflix members with similar tastes
32. The tech industry has only recently started to debate the ethics + blind spots of algorithms.
36. The language to describe privacy is improving. Hopefully, personalized recos will be the next to benefit from increased transparency.
37. “Privacy is a fundamental human right.”
38. Not sure if this is related to algorithms or privacy, but it’s a good sign.
39. Transparency is important for many reasons, including information asymmetry. At the risk of oversimplifying our central challenge: we know something the user doesn’t. So we need to share justifications and rationale in a transparent and trustworthy manner.
40. There’s a big power imbalance in big data.
42. Fair trade algorithms Organic algorithms Certified free of dark patterns
43. Jobs-to-be-done
44. Many personalized recommendations are: jobs-you-should-do-instead jobs-you-might-want-to-consider
45. It looks like you’re writing a letter. Many personalized recommendations are: jobs-you-should-do-instead jobs-you-might-want-to-consider
46. Mini-case study ⚗
47. A little bit about Kit
48. • April of 2016 — Shopify acquires Kit CRM • Kit is a virtual employee that works with merchants and delivers personalized recos that match their business • Kit uses conversational commerce to deliver those recos Hello from Kit
50. • Saying no to a reco is almost as valuable as saying yes • Designing for feedback is critical • Built around user success, not product retention A bit more about Kit
51. Design principles
52. “Design principles are a list of strongly-held opinions that an entire team agrees on. They force clarity and reduce ambiguity.” - Emmet Connolly, Intercom
54. • Aspirational • Transparent • Controllable • Challenging • Authentic Bibblio believes recos should be:
55. • Relevant — speak to user needs + be as specific as possible • Transparent — share the intent behind recommendation • Actionable — make next action clear; clarity over persuasion • Prudent — maintain user trust with high-confidence recos • Cohesive — share the results of taking a recommendation Our 1st draft principles:
56. Our principles have since evolved — we lost prudent, for example. And they’ll evolve again. Figuring out how recommendations work across the entire Shopify ecosystem is our next big challenge.
57. Beyond the sentence
58. Hello, {firstname}. It looks like you want to learn how Shopify combined {content strategy} and {design} to improve the {UX} of their personalized marketing recommendations.
59. My talk description pokes fun at Madlib sentences. Better known content formulas, they’re still the most common way to scale recos.
61. Airbnb recos use a content formula, but it works because it’s just plugging numbers into a sentence. Content formulas are convenient, but they aren’t always the best way to display a personalized recommendation.
63. Content formulas only go so far with detailed taxonomies.
65. Boost your chance of getting long-term bookings by 28% Want more longer-term bookings? Adding a 17% weekly discount for stays of 7 nights can boost your chance of getting these bookings. Add 17% discount No, thanks
66. Improve your long-term booking rate with a small discount Add 17% discount No, thanks Recommendation details Predicted outcome: 3 bookings over next 6 weeks Discount required: 17% Discount target: 7 night stays Outcome likelihood: 28%
67. • Develop content + experience principles for recos • Think beyond the sentence • Explore content + design implications of transparency Case study summary:
68. Conclusion
70. I gave Google a tough time today, but as I was putting this talk together, I found an article by Josh Lovejoy about the UX of AI. He lists design principles. He wants Google Clips to solve a real human need. We need more articles like this. It can be a struggle to create good user experiences in far less complicated circumstances. The only way to solve UX and machine learning challenges is to talk more often about our failures and successes.
71. Thanks! We have many worthwhile problems to solve at Shopify. Interested? I “recommend” you speak with me! Ryan Bigge
72. Appendix (
73. • https://medium.com/airbnb-engineering/how-we-deliver-insights-to-hosts-7d836520a38 • http://reallifemag.com/the-babysitters-club/ • http://nymag.com/vindicated/2016/10/clippy-didnt-just-annoy-you-he-changed-the-world.html • https://blog.intercom.com/principles-bot-design • http://www.bibblio.org/about • https://www.google.com/basepages/producttype/taxonomy.en-US.txt • https://design.google/library/ux-ai/ • https://thisisbucket.com/ References and resources