PyCon US 2020 - Decoding bias and narrative in competitive video games broadcasts with video analysis

PyCon US 2020 - https://us.pycon.org/2020/schedule/presentation/107/

With video game competitions (also known as eSports) being broadcast on online streams and regular TV stations, in-game video producers have to balance between making the game accessible to casual viewers, and packing enough action on screen to keep regular players interested.

What they choose to show on screen also define the narratives surrounding the competing teams, building hype around them and setting implicit expectations (for example what character class is shown the most).

In this talk we will walk through video analysis of professional Overwatch games to extract data, explore and validate a few hypotheses on what eSports broadcasters think viewers want to see.

Transcript

1. @kimadeline_m Decoding bias and narrative in competitive video games broadcasts with video analysis Kim-Adeline Miguel PyCon US 2020
2. @kimadeline_m All rights reserved All trademarks referenced herein are the properties of their respective owners. Blizzard Entertainment is a trademark or registered trademark of Blizzard Entertainment, Inc., in the U.S. and/or other countries. Overwatch and the Overwatch League are trademarks of Blizzard Entertainment, Inc., in the U.S. and/or other countries.
3. @kimadeline_m Outline • Why this talk • Building the system • Analyzing the findings • tl;dr
4. @kimadeline_m Why this talk Imagination is the essence of discovery.
5. @kimadeline_m Motivations Because I could :)
6. @kimadeline_m Motivations • Side project to learn Python • Think outside the (screen) box • Challenge assumptions with hypothesis-driven validation ✋ First, quick rundown on esports and Overwatch
7. @kimadeline_m Esports, e-sports, eSports • Wikipedia: “a form of sport competition using video games” • 1.1B hours watched in 2019 (top 5 most popular video games) • Online platforms (Twitch, YT Gaming), TV channels (ESPN, OGN) • Partnerships with traditional sports, ownership, endorsement deals, league/franchise systems Virtual sports but real
8. @kimadeline_m Overwatch according to its website • “Overwatch is a colorful team-based shooter game starring a diverse cast of powerful heroes. Travel the world, build a team, and contest objectives in exhilarating 6v6 combat.” • PC, Nintendo Switch, PlayStation 4, Xbox One
9. @kimadeline_m Anatomy of an Overwatch game • First-person shooter set in a near future on a fictionalized Earth (+ a colony on the Moon) • 2 teams of 6, that compete for objectives on different maps, multiple rounds • 32 character (“heroes”) to choose from, sorted in 3 roles: Tank, Damage, Support • Requirements: 2 heroes of each role per team
10. @kimadeline_m Anatomy of an Overwatch game Tank Shield/absorb damage Damage Pew pew Support Heal/enable damage
11. @kimadeline_m Anatomy of an Overwatch game Tank Shield/absorb damage Damage Pew pew Support Heal/enable damage
12. @kimadeline_m The Overwatch League () • City-based franchise league • Games streamed on YouTube Gaming • 1 game = multiple maps (first to 3 wins) • Roster up to 12 players, can swap between maps
13. @kimadeline_m Following the action In traditional sports the camera follows the ball, but there is no ball here. Who defines “the action”, and what is it?
14. @kimadeline_m Bias & hypotheses • Everybody wants to pew pew people would rather watch Damage heroes • Do people want to cheer for the underdog, or the team with the better win percentage? • Do people want to cheer for the winner?
15. @kimadeline_m Bias & hypotheses Is it what the viewers want, or what the producers think the viewers want?
16. @kimadeline_m Building the system It is time to act.
17. @kimadeline_m
18. @kimadeline_m
19. @kimadeline_m Process outline • Extract movie frames • Crop images • Send them to the OCR system • Store resulting data • Compute and display stats • Optional: Give a talk at PyCon about it
20. @kimadeline_m The Architectural Diagram Slide™ ☁ split crop store OCR ✨
21. @kimadeline_m Picking packages • Minimal setup (python -m pip install and go) • Reliable documentation • Relatively popular
22. @kimadeline_m Video splicing • FFmpeg: command-line-based processing of video and audio files • Python bindings: ffmpeg-python
23. @kimadeline_m Area cropping • Minimize noise • Pillow (fork from the Python Imaging Library)
24. @kimadeline_m Player name parsing • Samples: • First attempt: locally using tesseract-ocr • Second attempt: Azure Computer Vision services • REST API or library • Free tier: 20 calls/minute, 5000/month
25. @kimadeline_m Player name parsing using Azure • 2 types of calls: • async: Read API, supports printed and handwritten text • sync: OCR, only supports printed text • At least 720p video • Post-processing sanitizing
26. @kimadeline_m Data storage • Picking packages: minimal setup > documentation > popular • TinyDB: “lightweight document-oriented database optimized for your happiness” • No external dependencies • Documents stored as dicts
27. @kimadeline_m Data storage using TinyDB • Teams info (static data) • Game database structure: • 1 database per game • 1 table per map • 1 entry per frame "m3_volskaya": { "1": { "player": "SEOMINSOO", "frame": "0852" }, "2": { "player": "LEAVE", "frame": "0389" }, …
28. @kimadeline_m Data plotting & stats • Started with plotly.py • Browser-based graphing library • Wraps plotly.js • Next-level: Dash • plotly.js + React + Flask • Python web apps leveraging React components
29. @kimadeline_m Chart example
30. @kimadeline_m Retrospective • Dynamic player name area cropping • OCR limits: Automated sanitizing phase • Package choices (data storage, charting) • The art of beautiful charts
31. @kimadeline_m Analyzing the findings Imagine the worst possible outcome. Now… avoid that.
32. @kimadeline_m Dataset games from Week 10 (April 11-12, 2020):
33. @kimadeline_m Pre-game standings (wins-losses) • Hangzhou Spark: 2-2 • Chengdu Hunters: 1-3 • Vancouver Titans: 2-0 • Guangzhou Charge: 2-2 • Philadelphia Fusion: 6-1 • Paris Eternal: 5-2 • Houston Outlaws: 2-6 • Boston Uprising: 1-5 • Washington Justice: 2-6 • Dallas Fuel: 1-3 • Shanghai Dragons: 3-1 • Toronto Defiant: 3-4 • Atlanta Reign: 3-1 • San Francisco Shock: 2-2 • Los Angeles Gladiators: 2-2
34. @kimadeline_m Pre-game win percentage • Hangzhou Spark: 50% • Chengdu Hunters: 25% • Vancouver Titans: 100% • Guangzhou Charge: 50% • Philadelphia Fusion: 86% • Paris Eternal: 71% • Houston Outlaws: 25% • Boston Uprising: 17% • Washington Justice: 25% • Dallas Fuel: 25% • Shanghai Dragons: 75% • Toronto Defiant: 43% • Atlanta Reign: 75% • San Francisco Shock: 50% • Los Angeles Gladiators: 50%
35. @kimadeline_m Bias & hypotheses refresher • Everybody wants to pew pew people would rather watch Damage heroes • Do people want to cheer for the underdog, or the team with the better win percentage? • Do people want to cheer for the winner?
36. @kimadeline_m Results - roles • Average on-screen percentage per role: • Damage: 63.42% (49.39% - 80.52%) • Tank: 25.88% (15.06% - 41.03%) • Support: 10.7% (4.42% - 14.91%)
37. @kimadeline_m Results - teams • Average team difference: 9.71% (0.3% - 29.46%) • Games where the winning team had overall higher on- screen percentage: 8/10 • For most of the maps: 7/10 • Games where the team with the better win percentage had higher overall on-screen percentage: 5/10
38. @kimadeline_m Hypotheses vs results • Everybody wants to pew pew ✅ • Do people want to cheer for the underdog, or the team with the better win percentage? • Do people want to cheer for the winner? ✅
39. @kimadeline_m Never enough data • Other readable elements on the screen • Other camera angles:
40. @kimadeline_m Never enough data (2) • Per-map outcomes • Audio track • Sample size
41. @kimadeline_m tl;dr Catch phrase!
42. @kimadeline_m Takeaways https://twitter.com/rickhanlonii/status/1202664030621634561
43. @kimadeline_m Takeaways • It wasn’t so bad! Both the video analysis and hypothesis-driven development • Data can tell you anything and nothing • Only scratched the surface of what can be extracted from this footage Your turn now
44. @kimadeline_m github.com/kimadeline/overwatch-ocr twitter.com/kimadeline_m
45. @kimadeline_m ggwp everyone
46. @kimadeline_m Links – Esports and Overwatch • Viewership Statistics of 2019 Esports & Streaming: https://escharts.com/2019 • Overwatch official website: https://playoverwatch.com/ • The Overwatch League official website: https://overwatchleague.com
47. @kimadeline_m Links – Tools • Python extension for VS Code: https://marketplace.visualstudio.com/items?itemName=ms- python.python • FFmpeg API reference: https://kkroening.github.io/ffmpeg- python/ • TinyDB documentation: https://tinydb.readthedocs.io/en/latest/index.html • Plotly documentation: https://plotly.com/python/ • Dash documentation: https://dash.plotly.com/
48. @kimadeline_m Links – Tools (2) • Azure SDK multi-language quickstart: https://docs.microsoft.com/en-us/azure/cognitive- services/computer-vision/quickstarts-sdk/client-library • Azure SDK for Python Read API quickstart: https://docs.microsoft.com/en-us/azure/cognitive- services/Computer-vision/quickstarts/python-hand-text • Azure SDK for Python samples: https://github.com/Azure- Samples/cognitive-services-python-sdk-samples