Upgrade to Pro — share decks privately, control downloads, hide ads and more …

User Insights for Ultimate Personalization

User Insights for Ultimate Personalization

Edwin Chen

June 27, 2015

More Decks by Edwin Chen

Other Decks in Technology


  1. Life of a Quant ADS Do ads provide an effective

    ROI to advertisers? Do users become blind to too many ads? VIEWER Why do casual users turn core? How do users behave across different devices? CREATOR How do indie artists make it big? What makes a successful creator? INFRASTRUCTURE How do rebutters affect user behavior? Can we design better encoding algorithms? META Can we build a better experiment framework? Is watch time actually a good metric, and channel subscriptions a good goal?
  2. 2. Humans are sometimes better than machines. And an elite

    crowdsourcing proletariat is even better.
  3. What could we do with this information? 2. If Bob

    ever stops returning at 7pm, we have a concrete guess why: He probably ran out of sesamestreet-like content, so we should make an intervention to find him more.
  4. What could we do? 1. “After-school relaxation” could be a

    new habit we try to instill. Recommendations are more effective when we can explain why we’re giving them, so let’s find users in Alex’s demographic (male high-school teenagers bored after school) and sell this use case.
  5. What could we do? 2. Make it easy for Alex

    to navigate to his after- school gaming content by emailing or pushing it to him right when he needs it.
  6. This user has been fairly active for a while, but

    in April her activity kickstarted into action.
  7. This user has been fairly active for a while, but

    in April her activity kickstarted into action. What happened in April? She formed a crashcourse habit.
  8. This user didn’t start visiting everyday until April. What happened?

    That’s when his flashgameforever habit started.
  9. Why should we try to understand habits? 1. Habits are

    hard to break. Once ingrained, viewing rituals provide a repeatable reason to return to YouTube.
  10. Why should we try to understand habits? 1. Habits are

    hard to break. 2. Anticipate what users want. If we know you watch American Idol every Thursday, we should tune our recommendations accordingly. Imagine a YouTube Now experience.
  11. Why should we try to understand habits? 1. Habits are

    hard to break. 2. Anticipate what users want. 3. Better personalization. There’s no reason to recommend content we know you won’t be in the mood to watch.
  12. There are many problems we can work on. Since we’re

    quite different from the typical user, our intuition often fails. Winterfell is a tool to quickly investigate individual users and perform lightweight user research.
  13. An on-demand labor platform for finding thousands of people to

    work on tasks. We use an elite set of highly trained, highly curated workers. Hybrid
  14. The Video Inventory Analytics team found that, compared to the

    internal raters they had been using for a while, the Hybrid proletariat was: 7X cheaper 5X faster Higher quality: on 500 videos, their internal raters made 15 mistakes, compared to 3 mistakes by my workers on their first exposure to the task. Faster, Cheaper, Better
  15. We have forums, mailing lists, and chat rooms set up

    for easy feedback. Are your instructions unclear? Is your categorization task missing a useful category? Does your template contain a bug? They’ll let you know! Easy Communication
  16. The same workers work on the task each time, which

    means they can be more easily trained. Easy to Train
  17. Excellent Writers I sit down at the computer like, "What

    up? I got some big debt!" I'm so pumped about some tasks from my work desk Transcription on the downlow, it's so damn noisy Trying to make out people sayin', "Damn! That podcast's crazy!" Categorizin' ten layers deep, trying to get it right, Brain is startin' to keep me up all night Dreamin' about those tasks, pennies raining down Probably need a break, they're all I can think about (Hitssssssssssssssss....) But shit, it paid ninety-nine cents! (Grab it!) Acceptin' it, doing it', 'bout to go and submit it Passin' up on sunshine, can't believe it’s finished. Cheap ass labor, screw it, man! I'm waitin' on something' that pays Makin' my money and I'm hella happy that's a decent wage, bitch I'mma work your favorite hit, I'mma work your favorite hit No, for real - I'm work it - and you can't say nothin' about it (Thank you)
  18. Channel Annotations Can we manually improve tags of the top

    10K channels? Raters were given a channel, asked to watch videos from the channel to understand it, and then tasked with generating 5 free-form labels. Example Results - Crude humor - Do-it-yourself - Expert advice - Grammy winning - YouTube video commentary - Boy bands - Video game walkthroughs - Clips from TV shows - First person shooters - ....
  19. Creative Generation There is nothing more fun than playing video

    games, right? Actually, there is -- watching a Swedish guy play video games and narrate them! He's hilariously skittish, has a cute accent, and will make you feel superior when it comes to your gaming prowess. Seriously funny stuff. (for pewdiepie) Little monsters, gather ‘round as your mother has something to share with you. Through these videos, Gaga lays out who she is, through music, dance and interviews. Thought-provoking conversations, heart pumping pop and dramatic concert footage will all sweep you away to The Edge of Glory. (for LadyGagaVevo)
  20. Lightweight Surveys What makes users transition from casual to core?

    Causal inference is hard, and what we really want is to discover new ideas we never imagined. So why don’t we simply find a set of transitioned users and ask them?
  21. Lightweight Surveys “I have always been a big music fan,

    but recently I took up playing the guitar. Because of my new found passion (playing the guitar) my desire to watch concerts has increased. I started watching a whole lot of music festivals and concerts that are posted on Youtube and other music videos. I have spent a lot of time also watching guitar lessons on Youtube (from www.justinguitar.com).”
  22. Is matching users with great channels a sufficient goal? Maybe

    having enough time in the day is a limiting factor. Maybe we just don’t have the right content. Maybe users forget about the amazing channels they see.
  23. Many of our big launches (complete site redesign, channel recommendations

    in the feed, etc.) show a sizable increase in raw subscriptions, but modest changes to overall watch time. The watch time effect of projects like Fan Finder is also unclear. Is it because our algorithms aren’t perfect, and we just need to keep striving?
  24. Possible Effects 1. Multiplicative Engagement above and beyond activity on

    the channel itself. (Perhaps because users come to watch the new channel, and through recommendations stay to watch more.)
  25. Possible Effects 1. Multiplicative 2. Additive 3. Neutral The new

    channel replaces existing engagement, perhaps because users have a fixed amount of time (e.g., while on the train) to watch.
  26. Possible Effects 1. Multiplicative 2. Additive 3. Neutral 4. Negative

    Maybe users spend less time idly browsing once they have concrete channels they know to turn to, thereby decreasing their overall watch time.
  27. How can we understand these effects? Randomized experiments are the

    gold standard in establishing causality, but are often impossible to run. We can’t forcibly match users with channels we know they’ll love, because we don’t know which channels these are ourselves. We can’t randomly block users from their favorite channels either.
  28. Consider a user who uploads a new video every Wednesday.

    One month, he lets his subscribers know that he won’t be uploading any new videos for a few weeks, while he goes on vacation.
  29. How do his subscribers respond? Do they stop watching YouTube

    on Wednesdays, since his channel was the sole reason for their visits? Or is their activity relatively unaffected, since they only watch his content when it appears on the front page?
  30. What would happen if he starts uploading a new video

    every Friday? Do his subscribers start to visit then as well?
  31. Their overall watches (including watches on RayWilliamJohnson) also shifted to

    Wednesday and Saturday. However, this appeared to be an additive effect only. His fans did not start watching additional non- RayWilliamJohnson content on Wednesdays and Saturdays as well.
  32. Imagine you're fashion-clueless, and going shopping. How do you decide

    what to buy? If you've no idea how to dress, you probably find a shirt, some slacks, and you're done. But what if you had a personal stylist to find you the perfect outfit? Would you suddenly buy a handkerchief, a cardigan, and some cufflinks as well?
  33. What would the ultimate in personalization compel you to do?

    Let’s assign personal YouTube curators to a set of user participants, and see how their activity changes.
  34. If our problem is that users... • Don’t know about

    the wide variety of content that we have • Can’t find the right videos or channels • Don’t know how to use YouTube’s features …and if we fundamentally believe that solving these issues will lead to massive changes in watch time, then even a small curation experiment over 1000 users should exhibit a noticeable change.
  35. Sample Information “I'm interested in receiving curated content to help

    me cook more at home. I'd love to receive 4-5 instructional recipe videos on Sunday, for example, so I can buy the ingredients I need for the week and then try out one recipe a night. I really love California cuisine, like the type of stuff you find up in Wine Country. I also love variations of Italian, French, German, Spanish, Mediterranean...basically all of Eastern Europe :) My diet is reasonably healthy so I'd like to stick to fish, chicken, grains and veggies (plus any sauces and marinades) as much as possible but my guilty indulgences are always breads and cheeses and I'm always down to incorporate chorizo or a filet mignon.”
  36. Sample Curation “Laura is Italian. Laura makes home-cooked meals easy

    and fun. Need I say more? Gnocchi isn’t easy to make in 30 minutes (unless it’s frozen, natch), so instead I picked out a scratch fettuccini alfredo. Creamy, garlic-infused sauce envelops each strand of fettuccini like a loving hug from your Nonna. Laura even pronounces it correctly!”
  37. Subscriptions and fanships may not be the solution to 1B

    hours. From logs analysis and qualitative studies, instilling novel habits may prove fruitful. Habits classification is a large and difficult task, but luckily, we have an elite worker pool to help.
  38. Instead of “We think you may like pewdiepie...” “We think

    you may like gaming...” Perhaps “We think you may like some Minecraft to relax after school...”