FanGuard: A Tumblr Spoiler Filter (Insight Data Science)

FanGuard: a web-app using natural language processing to detect spoilers in Tumblr posts (without a "spoiler warning!"). Demo for Insight Data Science, SV-2016-A Session. http://www.fanguard.xyz/

Transcript

1. '"/(6"3%: A Smart Spoiler Filter for Tumblr Ruth Toner
2. • Tumblr: lots of very engaged fans, trading tons of images and text • SPOILERS can make people be very cautious or even avoid the site entirely [ Spoilers in here!] The Problem
3. • Goal: Can I create an intelligent spoiler filter to identify whether a given post is a spoiler, based on the language used in them? No spoiler tag! The Problem
4. Algorithm and Data Collect Posts from Tumblr Data Decide if post is about Star Wars Pre- Filter Decide if post is a Star Wars Spoiler Spoiler Filter • 25,000 posts per movie/show • Clean, stem, and tokenize • Does it contain Star Wars vocabulary words? • Features: Words in Tag, Words in Body (TF-IDF), number of words in post • Random Forest
5. Training and Validation Training: User-labeled posts labeled #Star-Wars and #Star-Wars-Spoilers Validation: Take out word “spoiler” from all posts • See if can predict whether a post was a spoiler! Regular Length: Median Words = 12 Mean Words = 39.9 Spoiler Length: Median Words = 34 Mean Words = 108.2
6. (SPOILER WARNING!)
7. Performance: Star Wars Spoilers Default Cut WCOUNT: # of words in text (T): Word used in tag All others = word used in body
8. = SPOILER
9. The Universal (Tumblr) Theory of Spoilers? (The Universal Language of spoilers is NSFW…)
10. About Me!
11. Backup
12. Avoiding Overtraining to Authors Need to remove posts from super-prolific authors…