Street Fighting Trend Research

PyData Berlin, July 26 2014

Transcript

1. STREET FIGHTING TREND RESEARCH BENEDIKT KOEHLER (@FURUKAMA) PYDATA BERLIN, 26 JULY 2014
2. 2 Copyright d.core GmbH, 2014 • Programming Mobile Applications in the 1990s • PhD in Sociology, research project on statistical visualization in the 2000s • Blog Research in the 2000s („metaroll“) • Social Media Intelligence @ethority • Now: Data Science in Media & Advertising @dcore_munich • Twitter: @furukama • Blog: beautifuldata.net • Mail: koehler@dcore.de • Github for this talk: https://github.com/furukama/pydata2014-berlin INTRODUCTION
3. 3 Copyright d.core GmbH, 2014 Source: http://drewconway.com/zia/2013/3/26/the-data-science-venn-diagram FOCUS OF THIS TALK
4. 4 Copyright d.core GmbH, 2014 • Rolling Stones -> Mathematics -> Data Science (Pete Skomoroch) • New use for data that is already there (e.g. geo-position in tweets -> movement) • New use for old methods (e.g. genome sequencing in advertising research) • A lot of Ad-hoc research (e.g. citation networks at political events) • Improvisations • Publishing your recipes (no more secret sauce) WHAT IS STREET FIGHTING TREND RESEARCH? http://en.wikipedia.org/wiki/File:Fightingmanstones.jpg
5. 5 Copyright d.core GmbH, 2014 • Predicting the future is more or less impossible (Yogi Berra theorem) • But:„The Past does not repeat itself, but it rhymes“ (Mark Twain) -> Telling stories about possible futures and developments • And: Predicting the present (Hyunyong Choi and Hal Varian) • Early indicators • Retrospect revisions • Data with little cost http://static.googleusercontent.com/media/www.google.com/de//googleblogs/pdfs/google_predicting_the_present.pdf PREDICTING THE PRESENT
6. 6 Copyright d.core GmbH, 2014 • Different layers of trends (Elina Hiltunen): • Megatrends: fundamental changes affecting many people in the world and lasting many years (e.g. ageing population, climate change, urbanization) • Trends: shorter and more local changes (e.g. messaging, blogging) • Wild Cards: sudden big events (e.g. 911) • Weak Signals: first signs of emerging change, often overlooked, not important – yet! Elina‘s dissertation: http://epub.lib.aalto.fi/pdf/diss/a365.pdf MEGATRENDS, TRENDS AND WEAK SIGNALS
7. 7 Copyright d.core GmbH, 2014 • Some sources for discovering weak signals we will be covering in this talk: Elina‘s dissertation: http://epub.lib.aalto.fi/pdf/diss/a365.pdf DISCOVERING WEAK SIGNALS
8. 8 Copyright d.core GmbH, 2014 • Delphi Method • Trendscouting aka Coolhunting • Scenarios • Visioning … “Google Cayce and you will find "coolhunter," and if you look closely you may see it suggested that she is a "sensitive" of some kind, a dowser in the world of global marketing. Though the truth, Damien would say, is closer to allergy, a morbid and sometimes violent reactivity to the semiotics of the marketplace.” COOLHUNTING IN LITERATURE
9. 9 Copyright d.core GmbH, 2014 • Ginsberg et. al: „Detecting influenza epidemics using search engine query data“ (2009) http://static.googleusercontent.com/media/research.google.com/de/archive/papers/detecting-influenza-epidemics.pdf GOOGLE FLU TRENDS
10. 10 Copyright d.core GmbH, 2014 • Kira Radinsky & Eric Horvitz: „Mining the Web to predict future events“ (2012) http://research.microsoft.com/en-us/um/people/horvitz/future_news_wsdm.pdf PREDICTING FUTURE EVENTS
11. 11 Copyright d.core GmbH, 2014 • Some things are harder to predict than others. • Precision = How many of the predicted events were right • Recall = How many of the reported events were correctly predicted. http://research.microsoft.com/en-us/um/people/horvitz/future_news_wsdm.pdf PREDICTING FUTURE EVENTS
12. 12 Copyright d.core GmbH, 2014 • Stanislav Nikolov: Nonparametric Timeseries Classification for Twitter trending topic detection http://snikolov.wordpress.com/2012/11/14/early-detection-of-twitter-trends/ NONPARAMETRIC TREND DETECTION
13. 13 Copyright d.core GmbH, 2014 • Peter A. Gloor (MIT): Coolhunting for Trends on the Web (2007) http://www.ickn.org/documents/Gloor_CTS07.pdf WEB COOLHUNTING
14. 14 Copyright d.core GmbH, 2014 GOOGLE TRENDS
15. 15 Copyright d.core GmbH, 2014 GOOGLE TRENDS
16. 16 Copyright d.core GmbH, 2014 TWITTER TRENDING TOPICS
17. 17 Copyright d.core GmbH, 2014 FOURSQUARE TRENDING LOCATIONS
18. 18 Copyright d.core GmbH, 2014 TECH / DATA TRENDS
19. 19 Copyright d.core GmbH, 2014 • Google Trends, Twitter Trending Topics, Foursquare Trending Locations are great, but … • No access to raw data • (Almost) no context • Closed source • Parameters can‘t be customized • Missing APIs • Not always the right topics • => Do It Yourself approach! QUANTITATIVE TREND RESEARCH
20. 20 Copyright d.core GmbH, 2014 • Example 1) Analysing Research Topics from the Web • Example 2) Mining Conference Proposals • Example 3) Identifying Trending Locations on Foursquare • Code at https://github.com/furukama/pydata2014-berlin EXAMPLES
21. QUESTIONS? @FURUKAMA