Tracking Live Wikipedia Edits

Realtime processing with Apache Kafka and Storm.
Also used, Apache Hadoop.

Transcript

1. TRACKING LIVE WIKIPEDIA CHANGES mo@andrewmo.com Insight Data Engineering
 Week 4 - January 2015 WIKIWATCH . ANDREWMO . COM
2. MOTIVATION • Raw dumps of Wikipedia data are available for analysis on a monthly basis, but…
 What about changes between these intervals? • Data Collection:
 Live edits for Wikimedia projects are broadcast to nearly 882 IRC channels • Goals: Collect, filter, format, transform and produce information from live edit data now data data why wait? what happened?
3. Realtime Telemetry wikiwatch . andrewmo . com
5. DATA PIPELINE ENGINEERING Capture + Fusion + Analysis Kafka Storm MySQL API IRC Map
 Reduce HBase HDFS
6. CAPTURE Up to 882 Simultaneous Channels ~660 events/min avg across all channels
7. INGEST Kafka + Hadoop Kafka #de logBot Topic Topic Topic #fr #en HDFS
8. –Andrew Mo We tried Spark Streaming. Scala, it’s not you - It’s me. Next sprint, maybe?
9. STREAM PROCESSING Multiple Topologies (10 sec, 1min, 10 min, 1 hr)
 Multiple Metrics
 (events, size, new pages, topics, users) Python + Storm (Pyleus) MySQL
10. API ACCESS Time Series Summary Metrics
 for Multiple Windows New Pages Detailed User Activity Detailed Topic Activity Top Topics, Top Users, Top Bots, etc
11. Thanks Apache Software Foundation Wikimedia Foundation Insight Data Science LinkedIn (Kafka) Twitter (Storm)
 Yelp (Pyleus)
12. ABOUT MO A Project Manager that Writes Code ! Worked at
 RAND Corporation
 Booz Allen Hamilton Studied at
 Pardee RAND Graduate School
 UC San Diego - Electrical Engineering Alphabet Soup
 PMP, PMI-ACP, CISSP, ISSEP, CSEP, CSEP-ACQ
 
 mo@andrewmo.com
 GitHub: https://github.com/moandcompany
 LinkedIn: http://linkedin.com/in/andrewmo
13. BONUS CHARTS
14. BATCH PROCESSING Map Reduce + Pig Hive MR Job Hadoop Streaming … Parser Log Pig Hive Spark HBase
15. DEDUPE IN THE BATCH LAYER • Parse all files and send to a Pig batch process • All log files go into a directory consumed by pig (‘LOAD’) • Apply ‘DISTINCT’ criteria to all rows in LOAD before computing metrics HDFS #en logBot #en #en logBot logBot
16. FIREHOSE Multiplex all sensors to a firehose topic Kafka #de logBot Omnichannel #fr #en logBot logBot
17. VELOCITY AND OUR NEXT SPRINT Sprint 1 (MVP Development) 
 18 Jan - 31 Jan 2015 Address the need + Simplify API-query elicitation and discovery Novel feature focus: Realtime Maximize common-code (Python) Sprint 2 (MVP Validation) Context Enrichment Features API enhancement Batch Integration NoSQL Optimization Preempt Technical Debt - Refactoring Velocity Chart
18. TECHNOLOGY TO EVALUATE • Presto • Luigi • Samza • Hive + Tez • Kafka on YARN (KOYA) • Kafka Security (Authentication) • Spark + Spark Streaming (1.2+ Python)
19. FAILURE MODES AND FAULT TOLERANCE • Implement Sensor Diversity (need de-dupe/false replay rejection) • The system cannot collect data if the IRC network is down • Define loss tolerance thresholds • Use logs as the ultimate source of truth