Deborah Hanus - Lights, camera, action! Scraping a great dataset to predict Oscar winners

Using Jupyter notebooks and scikit-learn, you’ll predict whether a movie is likely to [win an Oscar](http://oscarpredictor.github.io/) or be a box office hit. Together, we’ll step through the creation of an effective dataset: asking a question your data can answer, writing a web scraper, and answering those questions using nothing but Python libraries and data from the Internet.

https://us.pycon.org/2017/schedule/presentation/395/

Transcript

1. Lights Camera Action! Predicting Box Office hits using things you found on the Internet Deborah Hanus @deborahhanus
2. My Path
3. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
4. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
5. What factors drive movie revenue Image: unclaimedmoney.com
6. Not so good: Vague Will my movie be a box office hit? Good: Likelihood What is the likelihood that my movie will be a box office hit given that it has X features? Good: Correlation What attributes of a movie are correlated with box office success? Define a question you can answer
7. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
8. • Relevant • Structured • (Relatively) complete What is good data?
9. Where to find good data?
10. • Use an API • Write a web scraper • Get all the text • Make the text queryable How to get good data
11. • Make an HTTP request to get the HTML Writing a web scraper Requests Example
13. Writing a web scraper http://www.boxofficemojo.com/yearly/chart/? page=1&view=releasedate&view2=domestic&yr=%2 017.htm
16. • Make HTML queryable using BeautifulSoup & PyQuery Writing a web scraper
18. Right Click : View Source
24. • Rate limiting • API Keys • Selenium Common problems
25. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
26. Factors we can explore: • Movie budget • IMDB Rating • Power Studios • Opening Weekend • How many opening theaters • Seasonality • MPAA Rating Exploratory Data Analysis
27. Gross revenue vs. # opening theaters Exploratory Data Analysis ~3500
28. Gross revenue vs. Quality rating Exploratory Data Analysis No relationship :(
29. Gross revenue vs. Opening gross Exploratory Data Analysis Predictive
30. Multivariate regression Exploratory Data Analysis
31. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
32. • Budget helps (but only a little). • Timing is important. December is a great release date. • PG & G rated movies make more. • Money made in opening weekend is important. What did we find?
33. What does it take to win an Oscar?
34. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
35. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
36. What makes an Oscar winner? Image: superawesomevectors.com
37. Not so good: Vague Will this movie win an Oscar? Good: Likelihood What is the likelihood that this movie will win an Oscar given that it has X features? Good: Correlation What attributes of a movie are correlated with the movie winning an Oscar? Define a question you can answer
38. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
39. Not so good: Vague Will this movie win an Oscar? Good: Likelihood What is the likelihood that this movie will be a box office hit given that it has X features? Good: Correlation What attributes of a movie are correlated with the movie winning an Oscar? Define a question you can answer Best: Conditional correlation Given that a movie has been nominated for an Oscar, what attributes are correlated with winning?
40. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
41. Factors we can explore: • Movie nomination category • Thematic content (e.g. smoking, family, violence, war, father-son relationship) • Movie genre • Where the movie was made • When the movie debuts Exploratory Data Analysis
42. Exploratory Data Analysis Countries associated with winning Oscar movies
43. Exploratory Data Analysis Number of winning movies per month
44. Exploratory Data Analysis Ratio of winning films to films to nominated films by month
45. • Define a question you can answer • Acquire good data • Explore the data • Draw conclusions from the data How to build & use a great dataset
46. • Nominated films from Italy & Spain are likely to win • Films are more likely to win if they are released later in the year • Tone down the gore (unless it is a war film) • If you are nominated for best picture, your odds for winning are good. • If you are nominated for best cinematography, your odds are less good. What did we find?
47. • Defined an answerable question • Built a web scraper using Requests & BeautifulSoup • Explored the data • Used it to answer our questions What did we do?
48. Team
49. Building a scraper Requests - HTTP for Humans BeautifulSoup or PyQuery Analyzing data Jupyter SciKit Learn Statsmodel Example Projects http://oscarpredictor.github.io Where to go from here? Deborah Hanus @deborahhanus