Introduction to Big Data

Applications, Tools & Skills

Transcript

1. Big Data : Applications, Tools & Skills Eustache Diemert - Lead Scientist Bestofmedia / TechMediaNetwork 2013
2. Eustache Diemert • Lead Scientist, TechMediaNetwork ◦ Big Data Analytics ◦ Spam Fighting ◦ Monetization • Tech Lead, Yahoo! Web Search ◦ Machine Learned Ranking • contributor, Scikit-learn ◦ open source machine learning toolbox Speaker
3. Agenda Big What ? • Applications • Tools • Skills
4. Random Quotes: • “Big Data is like teenage sex: everyone thinks about it, but ...” – Anonymous (Big data consultant ?) • “Big Data = Hadoop” – Job Seeker (or Head Hunter) • “Big Data starts at 5 TB” – Hadoop Vendor (or IT Manager) • “For every 2°C up, ice cream sales go up by 2%” – CEO, Big Data Start-up (seeking funding probably) • “Big data really powers everything we do” – CTO, Big Data Company (pre-IPO ?) • “Big data will never give you big ideas” – Skeptic (maybe researcher) Big What ?
5. Big What ? Economic Reality • Big Data projects power whole sections of Digital Economy ◦ digital advertising : ~$40B in 2013 and growing by > 15% YoY • Brick & mortar companies to follow • Mostly prospective though… ◦ $1 spent generates $0.5 revenue for most projects ◦ $1 trillion growth potential ◦ 165,000 positions in “deep analytics”
6. Big What ? Research Reality • Sensors are everywhere ◦ Physics : CERN LHC → 25 PB data/year ◦ Social Sciences : web/mobile datasets → multi-PB ◦ Ecology : 1 month field study → 5 TB • Research on Data & Scale ◦ information retrieval, distributed computing, scaling machine learning etc ◦ tight Academia/Industry relationships
7. Applications
8. Apps: Web Search • Web Search at Yahoo! (~2007-2008) ◦ Distributed crawler ◦ Production grids ▪ processes html pages into indexable data ▪ computes inverted indexes ▪ MapReduce / Hadoop ▪ 30-50,000 machines ◦ Search Index ▪ 9B pages, 800 TB, compressed @11% ▪ 3 Tiers, different update/query policies ◦ Web Search : ▪ ~20,000 servers ▪ latency < 11ms @99%
9. Apps: Web Search ◦ Inverted Index Building in Map/Reduce ▪ Hadoop designed for this task • DFS + MR • code travels, not data
10. Apps: Out-of-core learning • ability to learn from data > RAM ◦ e.g. spam classification
11. Apps: Out-of-core learning • Anatomy of a large scale, OOC learner ◦ “big” training corpus ▪ split/replicate onto N machines ◦ extract features ▪ hashing trick : project to fixed size space ◦ learn N models ▪ e.g. SGD, RandomForests etc ◦ average N models for prediction ◦ recipe available here
12. Apps: Out-of-core learning • incremental algorithms • time : vectorization > learning
13. Tools
14. Tools • Open Source ◦ biggest Hadoop Contributor is still Yahoo! • Distributed Computing ◦ Hadoop as a platform ▪ whole galaxy : HBase, Impala, Avro, Pig, … ▪ YARN ◦ Successors / Competitors ▪ Storm, Spark, … • NoSQL Databases ◦ Cassandra, Mongo, Redis, Dynamo, … ◦ eventual consistency vs scalability ◦ column, document, key/value, graph, ...
15. Tools • Analytics ◦ R, python / scipy, … ◦ mostly “over hadoop” • Machine Learning ◦ Distributed modeling ◦ All-Reduce etc
16. Skills
17. Skills: Technology • Data Processing ◦ Line-oriented processing tools ▪ UNIX coreutils ▪ scripting (e.g. python) ▪ regular expressions • Statistical tools ◦ “at some scale, everything is statistical” ▪ e.g. Scipy, R • Software Engineering ◦ Distributed Processing Frameworks ▪ Hadoop + one of (Storm, Spark, ...)
18. Skills: Technology • Software Engineering ◦ Programming language ▪ object oriented (e.g. Java) ▪ functional (e.g. Python, Scala, Erlang, …) ◦ (Distributed) Databases ▪ landscape is moving fast ▪ hands-on experience a must
19. Skills: Science • Computer Science ◦ algorithms & data structures ◦ tame complexity at scale • Statistics ◦ averaging, binning, estimation theory ◦ information theory, sampling ◦ confidence intervals, tests • Machine Learning ◦ “prediction is power” ◦ statistical modeling
20. Take away Economy Science Technology Big Data
21. Questions ?