Live-Hack: Analyzing 7 years of Buzzwords (at Scale)

Talk given by Christoph Tavan at Berlin Buzzwords 2016, June 7.

https://www.berlinbuzzwords.de/session/live-hack-analyzing-7-years-buzzwords-scale

We're coming together for Berlin Buzzwords' 7th edition and over the course of the years a lot has changed in the Big Data Technology ecosystem. Once-hot buzzwords have vanished and new buzzwords arose.

While you would probably have written a MapReduce job in Java to crawl the web and analyze it on a massive scale this has now become much simpler with tools like Spark and Flink at hand.

I want to do a live coding session where I show that today it is possible to write a scalable web crawler and analytics tool which scrapes the past 6 years of Berlin Buzzwords (websites) and shows some interesting insights in the Big Data trends of the past 6 years. While I will run the tool on the very limited data set of the historical Berlin Buzzwords websites I want to highlight that it would in principle scale to crawl millions of websites and analyze petabytes of data.

Transcript

1. Live-Hack: Analyzing 7 years of Buzzwords (at Scale) Berlin Buzzwords – June 7, 2016 Christoph Tavan – @ctavan
2. Raise your hands... https://upload.wikimedia.org/wikipedia/commons/3/3f/La_ola_01.jpg
3. • CTO @ mbr targeting (Real- Time Bidding) • Attending Buzzwords since 2012 • First Talk @ Buzzwords 2015 We’re Hiring!
5. Buzzwords 2010 – 2016 !?!?!
8. So What Were The Actual Buzzwords? Scrape [2010-2016].berlinbuzzwords.de 1 2 Extract and Analyze Buzzwords … aaaaaand
9. So What Were The Actual Buzzwords? Scrape [2010-2016].berlinbuzzwords.de 1 2 Extract and Analyze Buzzwords … aaaaaand • Live! • Scalable! • In 30 Minutes...
10. So What Were The Actual Buzzwords? Scrape [2010-2016].berlinbuzzwords.de 1 2 Extract and Analyze Buzzwords
11. B) Scrape only relevant content (If you know what you’re looking for) Scraping – 2 Options A) Scrape all content, filter later (If you don’t know about the structure) +
12. Scraping – 2 Options + B) Scrape only relevant content (If you know what you’re looking for) http://scrapy.org/ A) Scrape all content, filter later (If you don’t know about the structure) http://nutch.apache.org/ Comprehensive Overview: http://www.ijser.org/researchpaper%5CComparison-of-Open-Source-Crawlers--A-Review.pdf
13. Scraping – 2 Options + B) Scrape only relevant content (If you know what you’re looking for) http://scrapy.org/ A) Scrape all content, filter later (If you don’t know about the structure) http://nutch.apache.org/ Comprehensive Overview: http://www.ijser.org/researchpaper%5CComparison-of-Open-Source-Crawlers--A-Review.pdf
14. Scrapy
15. Problems During Scraping No Content-Type Header for 2010-2012: -> Scrapy won’t parse! ???
16. ✔ Is a Page a Session Page? .date-display-single ? ✔ ✘ ✘ ✔ ✔ ✔
17. ✔ Is a Page a Session Page? .date-display-single ? ✔ ✘ ✘ ✔ ✔ ✔ 2012 and 2013: .field-field-session-slot else: .date-display-single Is a Page a Session Page?
18. Who’s the Speaker? .field-field-speaker ? ✔ ✔ ✘ ✘ ✘ ✘ ✘
19. Who’s the Speaker? .field-field-speaker ? ✔ ✔ ✘ ✘ ✘ ✘ ✘ 2010 and 2011: .field-field-speaker a 2012: .field-field-indiv-speakers a 2013: .field-field-speakers a 2014-2016: .field-name-field-session-speaker a Who’s the Speaker?
20. 2010 – 2013: #main p 2014 – 2016: article p And Where’s the Session Abstract?
21. https://i.ytimg.com/vi/Nt-dP2AiOkQ/maxresdefault.jpg Conclusion: Scraping is hard … … even in 2016!
22. So What Were The Actual Buzzwords? Scrape [2010-2016].berlinbuzzwords.de 1 2 Extract and Analyze Buzzwords
23. Interactive Notebooks • Commercial: ◦ https://databricks.com/ ◦ https://www.qubole.com/ ◦ http://www.cloudwick.com/ ◦ Review: http://www.infoworld.com/article/3068519/artificial- intelligence/review-6-machine-learning-clouds.html • Open Source: ◦ http://jupyter.org/ ◦ https://zeppelin.incubator.apache.org/ ◦ http://spark-notebook.io/
25. So What Were The Actual Buzzwords? Scrape [2010-2016].berlinbuzzwords.de 1 2 Extract and Analyze Buzzwords … aaaaaand • Live! ✔ • Scalable! ✔ (at least analysis) • In 30 Minutes… ✔
26. MapReduce WordCount (not in 30 minutes…) https://hadoop.apache.org/docs/r1.0.4/mapred_tutorial.html#Example%3A+WordCount+v1.0
27. Conclusion
28. https://i.ytimg.com/vi/Nt-dP2AiOkQ/maxresdefault.jpg Scraping is hard … … even in 2016!
29. Berlin Buzzwords #1 Top Speaker!
30. More Berlin Buzzwords Top Speakers! Grant Ingersoll Uwe Schindler Eric Evans
31. Flink Took Over Spark this Year!
32. The Age of Streaming is There! (Who would have guessed?!? )
33. Thank You So Much! Questions? Twitter: @ctavan https://github.com/ctavan/bbuzz2016 christoph@mbr-targeting.com https://mbr-targeting.com