What Does The Data Say? User Insights for Ultimate Personalization Edwin Chen

Life of a Quant ADS Do ads provide an effective ROI to advertisers? Do users become blind to too many ads? VIEWER Why do casual users turn core? How do users behave across different devices? CREATOR How do indie artists make it big? What makes a successful creator? INFRASTRUCTURE How do rebutters affect user behavior? Can we design better encoding algorithms? META Can we build a better experiment framework? Is watch time actually a good metric, and channel subscriptions a good goal?

3 TL; FELL ASLEEP

1. Users have a variety of time-based habits we should ingrain.

2. Humans are sometimes better than machines. And an elite crowdsourcing proletariat is even better.

3. What’s the key to increasing watch time? Fanships? Good content? User knowledge?

7 FRIDAY, FRIDAY User Habits for YouTube Now

Imagine Bob tunes in to sesamestreet every night at 7pm.

What could we do with this information? 1. Stop recommending Blue’s Clues during the day.

What could we do with this information? 2. If Bob ever stops returning at 7pm, we have a concrete guess why: He probably ran out of sesamestreet-like content, so we should make an intervention to find him more.

Imagine Alex jams out to gaming channels every day after school on his phone.

What could we do? 1. “After-school relaxation” could be a new habit we try to instill. Recommendations are more effective when we can explain why we’re giving them, so let’s find users in Alex’s demographic (male high-school teenagers bored after school) and sell this use case.

What could we do? 2. Make it easy for Alex to navigate to his after- school gaming content by emailing or pushing it to him right when he needs it.

What rituals do users have?

A user who listens to Rebecca Black every Friday.

A user who searches for different TV shows, depending on the day of week.

A user who searches for the “Rosemary and Thyme” TV show every midnight.

A user who watches religious content in the morning, but entertainment at night.

A user who falls asleep to YouTube TV.

What’s the effect of a new habit? Ongoing work, but...

This user has been fairly active for a while, but in April her activity kickstarted into action.

This user has been fairly active for a while, but in April her activity kickstarted into action. What happened in April? She formed a crashcourse habit.

This user didn’t start visiting everyday until April.

This user didn’t start visiting everyday until April. What happened? That’s when his flashgameforever habit started.

Why should we try to understand habits? 1. Habits are hard to break. Once ingrained, viewing rituals provide a repeatable reason to return to YouTube.

Why should we try to understand habits? 1. Habits are hard to break. 2. Anticipate what users want. If we know you watch American Idol every Thursday, we should tune our recommendations accordingly. Imagine a YouTube Now experience.

Why should we try to understand habits? 1. Habits are hard to break. 2. Anticipate what users want. 3. Better personalization. There’s no reason to recommend content we know you won’t be in the mood to watch.

Winterfell

There are many problems we can work on. Since we’re quite different from the typical user, our intuition often fails. Winterfell is a tool to quickly investigate individual users and perform lightweight user research.

30 OH, THE HUMANITY Applications of Elite Crowdsourcing

An on-demand labor platform for finding thousands of people to work on tasks. We use an elite set of highly trained, highly curated workers. Hybrid

Advantages over other internal platforms

The Video Inventory Analytics team found that, compared to the internal raters they had been using for a while, the Hybrid proletariat was: 7X cheaper 5X faster Higher quality: on 500 videos, their internal raters made 15 mistakes, compared to 3 mistakes by my workers on their first exposure to the task. Faster, Cheaper, Better

We have forums, mailing lists, and chat rooms set up for easy feedback. Are your instructions unclear? Is your categorization task missing a useful category? Does your template contain a bug? They’ll let you know! Easy Communication

The same workers work on the task each time, which means they can be more easily trained. Easy to Train

Excellent Writers I sit down at the computer like, "What up? I got some big debt!" I'm so pumped about some tasks from my work desk Transcription on the downlow, it's so damn noisy Trying to make out people sayin', "Damn! That podcast's crazy!" Categorizin' ten layers deep, trying to get it right, Brain is startin' to keep me up all night Dreamin' about those tasks, pennies raining down Probably need a break, they're all I can think about (Hitssssssssssssssss....) But shit, it paid ninety-nine cents! (Grab it!) Acceptin' it, doing it', 'bout to go and submit it Passin' up on sunshine, can't believe it’s finished. Cheap ass labor, screw it, man! I'm waitin' on something' that pays Makin' my money and I'm hella happy that's a decent wage, bitch I'mma work your favorite hit, I'mma work your favorite hit No, for real - I'm work it - and you can't say nothin' about it (Thank you)

What can you do with an elite proletariat?

Categorization++ Categorize the following music-related search query.

Channel Annotations Can we manually improve tags of the top 10K channels? Raters were given a channel, asked to watch videos from the channel to understand it, and then tasked with generating 5 free-form labels. Example Results - Crude humor - Do-it-yourself - Expert advice - Grammy winning - YouTube video commentary - Boy bands - Video game walkthroughs - Clips from TV shows - First person shooters - ....

Creative Generation pewdiepie has a pretty boring channel description. Can we crowdsource a better one?

Creative Generation There is nothing more fun than playing video games, right? Actually, there is -- watching a Swedish guy play video games and narrate them! He's hilariously skittish, has a cute accent, and will make you feel superior when it comes to your gaming prowess. Seriously funny stuff. (for pewdiepie) Little monsters, gather ‘round as your mother has something to share with you. Through these videos, Gaga lays out who she is, through music, dance and interviews. Thought-provoking conversations, heart pumping pop and dramatic concert footage will all sweep you away to The Edge of Glory. (for LadyGagaVevo)

Lightweight Surveys What makes users transition from casual to core? Causal inference is hard, and what we really want is to discover new ideas we never imagined. So why don’t we simply find a set of transitioned users and ask them?

Lightweight Surveys “I have always been a big music fan, but recently I took up playing the guitar. Because of my new found passion (playing the guitar) my desire to watch concerts has increased. I started watching a whole lot of music festivals and concerts that are posted on Youtube and other music videos. I have spent a lot of time also watching guitar lessons on Youtube (from www.justinguitar.com).”

44 LOVE IN THE TIME OF CHANNELS Is Love All We Need?

Is matching users with great channels a sufficient goal? Maybe having enough time in the day is a limiting factor. Maybe we just don’t have the right content. Maybe users forget about the amazing channels they see.

Many of our big launches (complete site redesign, channel recommendations in the feed, etc.) show a sizable increase in raw subscriptions, but modest changes to overall watch time. The watch time effect of projects like Fan Finder is also unclear. Is it because our algorithms aren’t perfect, and we just need to keep striving?

What would happen if we found users the perfect match?

Possible Effects 1. Multiplicative Engagement above and beyond activity on the channel itself. (Perhaps because users come to watch the new channel, and through recommendations stay to watch more.)

Possible Effects 1. Multiplicative 2. Additive Activity increases, but on the new channel alone.

Possible Effects 1. Multiplicative 2. Additive 3. Neutral The new channel replaces existing engagement, perhaps because users have a fixed amount of time (e.g., while on the train) to watch.

Possible Effects 1. Multiplicative 2. Additive 3. Neutral 4. Negative Maybe users spend less time idly browsing once they have concrete channels they know to turn to, thereby decreasing their overall watch time.

How can we understand these effects? Randomized experiments are the gold standard in establishing causality, but are often impossible to run. We can’t forcibly match users with channels we know they’ll love, because we don’t know which channels these are ourselves. We can’t randomly block users from their favorite channels either.

Enter Natural Experiments

Consider a user who uploads a new video every Wednesday. One month, he lets his subscribers know that he won’t be uploading any new videos for a few weeks, while he goes on vacation.

How do his subscribers respond? Do they stop watching YouTube on Wednesdays, since his channel was the sole reason for their visits? Or is their activity relatively unaffected, since they only watch his content when it appears on the front page?

What would happen if he starts uploading a new video every Friday? Do his subscribers start to visit then as well?

RayWilliamJohnson Between 2011 and 2012, RayWilliamJohnson switched from uploading videos on Tuesday and Friday, to Wednesday and Saturday.

RayWilliamJohnson How did his shift change the activity of his fans?

As expected, his fans also shifted to watching his videos on Wednesday and Saturday.

Their overall watches (including watches on RayWilliamJohnson) also shifted to Wednesday and Saturday.

Their overall watches (including watches on RayWilliamJohnson) also shifted to Wednesday and Saturday. However, this appeared to be an additive effect only. His fans did not start watching additional non- RayWilliamJohnson content on Wednesdays and Saturdays as well.

Personal Curation Pilot

Imagine you're fashion-clueless, and going shopping. How do you decide what to buy?

Imagine you're fashion-clueless, and going shopping. How do you decide what to buy? If you've no idea how to dress, you probably find a shirt, some slacks, and you're done. But what if you had a personal stylist to find you the perfect outfit? Would you suddenly buy a handkerchief, a cardigan, and some cufflinks as well?

What would the ultimate in personalization compel you to do? Let’s assign personal YouTube curators to a set of user participants, and see how their activity changes.

If our problem is that users... • Don’t know about the wide variety of content that we have • Can’t find the right videos or channels • Don’t know how to use YouTube’s features …and if we fundamentally believe that solving these issues will lead to massive changes in watch time, then even a small curation experiment over 1000 users should exhibit a noticeable change.

No content

Sample Information “I'm interested in receiving curated content to help me cook more at home. I'd love to receive 4-5 instructional recipe videos on Sunday, for example, so I can buy the ingredients I need for the week and then try out one recipe a night. I really love California cuisine, like the type of stuff you find up in Wine Country. I also love variations of Italian, French, German, Spanish, Mediterranean...basically all of Eastern Europe :) My diet is reasonably healthy so I'd like to stick to fish, chicken, grains and veggies (plus any sauces and marinades) as much as possible but my guilty indulgences are always breads and cheeses and I'm always down to incorporate chorizo or a filet mignon.”

Sample Curation “Laura is Italian. Laura makes home-cooked meals easy and fun. Need I say more? Gnocchi isn’t easy to make in 30 minutes (unless it’s frozen, natch), so instead I picked out a scratch fettuccini alfredo. Creamy, garlic-infused sauce envelops each strand of fettuccini like a loving hug from your Nonna. Laura even pronounces it correctly!”

70 END RECAP

Subscriptions and fanships may not be the solution to 1B hours. From logs analysis and qualitative studies, instilling novel habits may prove fruitful. Habits classification is a large and difficult task, but luckily, we have an elite worker pool to help.

Instead of “We think you may like pewdiepie...” “We think you may like gaming...” Perhaps “We think you may like some Minecraft to relax after school...”

Questions?