Drinking from the Firehose

Transcript

1. Drinking from the Firehose Real-Time Metrics Samantha Quiñones
2. @ieatkillerbees http://samanthaquinones.com
4. “How would you let editors test how well different headlines perform for the same piece of content?”
5. Areas of Interest • Quantified Application Performance • Perceived Application Performance (User Experience) • User Behavior
6. Quantified Application Performance
7. • Are requests being handled efficiently? • Are adequate resources available at each layer of the stack? • Is cache being utilized in an efficient manner?
8. CPU Time Per Request
9. Stack Performance
10. Perceived User Performance
11. • How long does it take for a page to be “ready” for the user? • Is the page responsive to user input? • Does the page require an excessive number of requests to complete?
13. Delay Perception <100ms Instantaneous <300ms Perceptible Delay <1000ms System “Working” <10000ms System “Slow” >10000ms System “Down” Source: O’Reilly Media
14. Effects of Latency Amazon Sales: -1% sales per 100ms increased latency Sales (USD) 0 500 1000 1500 2000 Seconds of Latency 0 1 2 3 5 6 7 8 9 10 11 12 13 14 15 S = S - (msL*1%) Linden, G. (2006, December 3). Make Data Useful. Data Mining (CS345). Lecture conducted from Stanford University, Stanford, CA.
15. Effects of Latency Google Search Experiment (4-6 Weeks) % Fewer Searches per Day -1 -0.75 -0.5 -0.25 0 Milliseconds of Additional Latency 50 100 200 400 Schurman, E., Brutlag, J. (2009, June 23). The User and Business Impact of Server Delays, Additional Bytes, and HTTP Chunking in Web Search. O'Reilly Velocity. Lecture conducted from O'Reilly Media, San Jose, CA.
16. Interpreting Web Profiler Information
18. User Behavior
19. Measuring User Behavior • Application path • Use patterns • Mouse & attention tracking
20. Multivariate Testing • Sort all users in to groups • 1 control group receives unaltered content • 1 or more groups receive altered content • Measure behavioral statistics (CTR, abandon rate, time on page, scroll depth) for each group
21. Managing Big Data
22. How big is big?
23. 1,300,000,000,000 events per DAY
24. ~40 datapoints per EVENT
25. ~15,000 records per SECOND
26. ~600,000 datapoints Containing
27. 25 megabytes / second At a rate up to
28. Collector Collector Collector Collector Collector Collector Collector Collector Collector Collector Collector Collector Rabbit MQ Farm
29. Rabbit MQ Farm Hadoop
30. Hadoop • Framework for distributed storage and processing of data • Designed to make managing very large datasets simple with… • Well-documented, open-source, common libraries • Optimizing for commodity hardware
31. Hadoop Distributed File System • Modeled after Google File System • Stores logical files across multiple systems • Rack-aware • No read-write concurrency
32. MapReduce • Framework for massively parallel data processing tasks
33. Map <?php $document = "I'm a little teapot short and stout here is my handle here is my spout"; /** * Outputs: [0,0,0,0,0,0,0,0,1,0,0,0,1,0,0] */ function map($target_word, $document) { return array_map( function ($word) use ($target_word) { if ($word === $target_word) { return 1; } return 0; }, preg_split('/\s+/', $document) ); } echo json_encode(map("is", $document)) . PHP_EOL;
34. Reduce <?php $data = [0,0,0,0,0,0,0,0,1,0,0,0,1,0,0]; /** * Outputs: 2 */ function reduce($data) { return array_reduce( $data, function ($count, $value) { return $count + $value; } ); } echo reduce($data) . PHP_EOL;
35. Hadoop Limitations • Hadoop jobs are batched and take significant time to run • Data may not be available for 1+ hours after collection
36. “How would you let editors test how well different headlines perform for the same piece of content?”
37. Consider Shelf-life • Most articles are relevant for < 24 hours • Interest peaks < 3 hours
38. Real-Time Pipelines
39. Collector Collector Collector Collector Collector Collector Collector Collector Collector Collector Collector Collector Rabbit MQ Farm Collector Collector Collector Streamer Collector Collector Collector Streamer Collector Collector Collector Streamer
41. Version 1 (PoC) Collector Collector Collector Streamer Collector Collector Collector Receiver Collector Collector Collector StatsD Cluster ElasticSearch
43. this.visit = function(record) { if (record.userAgent) { var parser = new UAParser(); parser.setUA(record.userAgent); var user_agent = parser.getResult(); return { user_agent: user_agent } } return {}; };
45. Findings • Max throughput per collector: 300 events/second • ~70 receivers needed for prod • StatsD key format creates data redundancy and reduced data richness
46. Version 1 (PoC) Collector Collector Collector Streamer Collector Collector Collector Receiver Collector Collector Collector StatsD Cluster ElasticSearch
47. Transits & Terminals • Transits - Short-term, in-memory, volatile storage for data with a life-span up to a few seconds • Terminals - Destinations for data that either store, abandon, or transmit
48. An efficient real-time data pathway consists of a network of transits and terminals, where no single node acts as both a transit and a terminal at the same time.
49. StatsD • Acts as a transit, taking data and passing it along… • BUT • Acts as a terminal, aggregating keys in memory and becoming a transit after a time or buffer threshold.
50. Version 2 Collector Collector Collector Streamer Collector Collector Collector Receiver ElasticSearch RabbitMQ
52. Version 2 • Eliminated eventing and improved performance • Replaced StatsD with RabbitMQ • Data records are kept together • No longer works with Kibana (sadface)
53. RabbitMQ • Lightweight message broker • Allows complex message routing without application-level logic • Can buffer 90-120 seconds of traffic
54. while (buffer.length > 0) { var char = buffer.shift(); if ('\n' === char) { queue.push(new Buffer(outbuf.join(''))); continue; } outbuf.push(char); } var i = 0; var tBuf = buffer.slice(); while (i < buffer.length) { var char = tBuf[i++]; if ('\n' === char) { queue.push(new Buffer(outbuf.join(''))); } outbuf.push(char); }
55. Findings • Max throughput per collector: 600 events/second • ~35 receivers needed for prod • Micro-optimized code became increasingly brittle and hard to maintain as custom logic was needed for every edge case
56. Version 2 Collector Collector Collector Streamer Collector Collector Collector Receiver ElasticSearch RabbitMQ
57. Need to Get Serious • Very high throughput • Multi-threaded worker pool with large memory buffers • Static & dynamic optimization • Efficient memory management for extremely volatile in-memory data • Eliminate any processing overhead. Receiver must be a Transit
58. And also… • Not GoLang (because no one on the team is familiar with it) • Not Rust (because no one on the team wants to be familiar with it) • Not C (because C)
60. mfw java :(
61. Why Java? • Solid static & dynamic analysis and optimizations in the S2BC & JIT compilers • Clients for the stuff I needed to talk to • Well-supported within AOL & within my team
62. Version 3 Collector Collector Collector Streamer Collector Collector Collector Receiver ElasticSearch RabbitMQ Collector Collector Collector Processor/ Router
64. public class StreamReader { private static final Logger logger = Logger.getLogger(StreamReader.class.getName()); private StreamerQueue queue = new StreamerQueue(); private StreamProcessor processor; private List<StreamReader.BeaconWorkerThread> workerThreads = new ArrayList(); private RtStreamerClient client; public StreamReader(String streamerURI, AmqpClient amqpClient, String appID, String tpcFltrs, String rfFltrs, String bt) { ArrayList queueList = new ArrayList(); this.processor = new StreamProcessor(amqpClient); byte numThreads = 8; for(int i = 0; i < numThreads; ++i) { StreamReader.BeaconWorkerThread worker = new StreamReader.BeaconWorkerThread(); this.workerThreads.add(worker); worker.start(); } queueList.add(this.queue); this.client = new RtStreamerClient(streamerURI, appID, tpcFltrs, rfFltrs, bt, queueList); } }
65. public class StreamProcessor { private static final Logger logger = Logger.getLogger(StreamProcessor.class.getName()); private AmqpClient amqpClient; public StreamProcessor(AmqpClient amqpClient) { this.amqpClient = amqpClient; } public void send(String data) throws Exception { this.amqpClient.send(data.getBytes()); logger.debug("Sent event " + data + " to AMQP"); } }
66. Queue Queue Queue Queue Queue Queue Queue Queue Queue Queue Queue Network Input Network Output Linked List Queues
67. Findings • Max throughput per collector: 2600 events/second • ~10 receivers needed for prod
69. Why ElasticSearch • Open-source Lucene search engine • Highly-distributed storage engine • Clusters nicely • Built-in aggregations like whoa
70. Aggregations • Geographic Boxing & Radius Grouping • Time-Series • Histograms • Min/Max/Avg Statistical Evaluation • MapReduce (coming soon!)
71. • How many users viewed my post on an android tablet in portrait mode within 10 miles of Denton, TX? • What is the average time from start of page-load to first click for readers on linux desktops between 3am and 5am? • Given two sets of link texts, which has the higher CTR for a randomized sample of readers on tablet devices?
72. Browser to Browser in < 5 seconds
73. But wait… Is that “real-time”?
74. Real-Time for Real • Live analysis of data as it is collected • Active visualization of very short-term trends in data
75. Potential Problems • Small sample sizes for new datasets / small analysis windows • Data volumes too high for end-user comprehension • Data volumes too high for end-user hardware/network connections
76. Version 4 Collector Collector Collector Streamer Collector Collector Collector Receiver ElasticSearch RabbitMQ Collector Collector Collector Processor/ Router Websocket Server
78. D3JS • Open-source data visualization library written in JavaScript
79. function plot(point) { var points = svg.selectAll("circle") .data([point], function(d) { return d.id; }); points.enter() .append("circle") .attr("cx", function (d) { return projection([parseInt(d.location.geopoint.lon), parseInt(d.location.geopoint.lat)])[0] }) .attr("cy", function (d) { return projection([parseInt(d.location.geopoint.lon), parseInt(d.location.geopoint.lat)])[1] }) .attr("r", function (d) { return 1; }) .style('fill', 'red') .style('fill-opacity', 1) .style('stroke', 'red') .style('stroke-width', '0.5px') .style('stroke-opacity', 1) .transition() .duration(10000) .style('fill-opacity', 0) .style('stroke-opacity', 0) .attr('r', '32px').remove(); } var buffer = []; var socket = io(); socket.on('geopoint', function(point) { if (point.location.geopoint) { plot(point); } });
82. By the way… xn = x + (r * COS(2π * n / v)) yn = y + (r * COS(2π * n / v)) where n = ordinal of vertex and where v = number of vertices and x,y = center of the polygon
83. var views = 0; var socket = io(); socket.on('pageview', function(point) { views++; }); function tick() { data.push(views); views = 0; path .attr("d", line) .attr("transform", null) .transition() .duration(500) .ease("linear") .attr("transform", "translate(" + x(0) + ",0)") .each("end", tick); data.shift(); } tick();
84. Pageview Heartbeat
86. Receiver Layer Receiver Buffer/Transit Processing & Routing Layer Processing & Routing Transit Storage Engine End-User Consumable Queues Layers are • Geographically decoupled • Capable of independent scaling • Fully encapsulated with no cross- layer dependencies
87. Interfaces Input Stream (Java) Routing (node.js) Filtering (node.js) Aggregation (PHP) Visualization (D3JS) MV Testing (PHP)
88. Languages & Tools Rabbit MQ Hadoop Elastic Search PHP JS (node) JS (D3) Java MySQL
89. Where are we Now? • It took 8 months to build a rock-solid data pipeline • Entry points from: • User data collectors • Application code
90. That was the easy part.
91. What’s next?
92. • Live debugging & runtime profiling • Embeddable visualizations • On-demand stream filters
93. ???
94. @ieatkillerbees http://samanthaquinones.com