Front-to-Back: Architecting your data reporting and visualization stack

A Fastly.com warstory that details the trials, tribulations, and lessons learned from building a custom stats and reporting system. Talk given at SctolandJS 2013 (http://scotlandjs.com/).

Transcript

1. Front-to-back Architecting your data reporting and visualization stack Ryan Sandor Richards (@rsandor) ryan@fastly.com Tuesday, May 14, 2013
2. Expectations Tuesday, May 14, 2013
3. Expectations Technical, Tutorial Tuesday, May 14, 2013
4. Expectations Technical, Tutorial Architectural, Anecdotal Tuesday, May 14, 2013
5. • Content Delivery Network • Based on Varnish • Datacenters around the world • 10^10 requests / day • https://www.fastly.com What is Fastly? Tuesday, May 14, 2013
6. Stats Architecture Backend Middle-end Front-end Collection & Storage Querying & Publishing Visualization & Reporting Tuesday, May 14, 2013
7. Real-time and Historical Tuesday, May 14, 2013
8. San Francisco, 2011 Real-time Tuesday, May 14, 2013
9. ARTUR BERGMAN (CEO, FASTLY) Tuesday, May 14, 2013
10. ARTUR BERGMAN (VIKING, FASTLY) goo.gl/U2iCX Tuesday, May 14, 2013
11. “Dude...” Tuesday, May 14, 2013
12. “Sup?” Tuesday, May 14, 2013
13. “I want this...” Tuesday, May 14, 2013
14. “Varnish Stats...” Tuesday, May 14, 2013
15. “...” Tuesday, May 14, 2013
16. “ON THE WEB AND PRETTY!” Tuesday, May 14, 2013
17. “Oh I’ll use d3! how do we get the data?” Tuesday, May 14, 2013
18. “ØMQ and Socket.IO!” Tuesday, May 14, 2013
19. Like, 5 minutes later... Tuesday, May 14, 2013
20. Cache Aggregator Cache Cache Socket.IO Client Client Client Tuesday, May 14, 2013
21. Cache Aggregator Cache Cache Socket.IO Client Client Client Latency Memory Leaks Unreliable Tuesday, May 14, 2013
22. Tuesday, May 14, 2013
23. ØMQ IS NOT DESIGNED FOR CROSS DATACENTER APPLICATIONS Tuesday, May 14, 2013
24. • You can’t • Queuing and waiting before publishing (eventual convergence) • Backfill in the Visualizations Fixing Latency Tuesday, May 14, 2013
25. Fixing ØMQ and Leaks • Caused by bindings and node • Replaced ØMQ with HTTP • Rewritten in Perl (GASP!) Tuesday, May 14, 2013
26. Fixing Socket.IO • Quit Using It • Used a third-party expert: PubNub Tuesday, May 14, 2013
27. d3 is sweet, but... • Didn’t work well for real-time vis (this is before cubism, etc.) • I had very little experience with it • Ended up using HTML5 Canvas Tuesday, May 14, 2013
28. https://www.fastly.com/demo Results Tuesday, May 14, 2013
29. https://www.fastly.com/demo Results Tuesday, May 14, 2013
30. •USE WHAT YOU KNOW... • Perl vs. Node • HTTP vs. ØMQ • Canvas vs. d3 •IF YOU DON’T KNOW... • PubNub over Socket.IO Lessons Tuesday, May 14, 2013
31. Ask good questions up front Tuesday, May 14, 2013
32. San Francisco, 2012 Historical Stats Tuesday, May 14, 2013
33. “RYAN! STATS HISTORICAL NOW” Tuesday, May 14, 2013
34. “Oohkay, so how are we...” Tuesday, May 14, 2013
35. “BUSY. YOU DO IT. RUNNING COMPANY!” Tuesday, May 14, 2013
36. “...” Tuesday, May 14, 2013
37. We Knew Nothing • What did our users need? • What did we need internally? • Should we have an API? • How fast is our data growing? • What’s visualizations are needed? Tuesday, May 14, 2013
38. But we had a plan... • A month of investigative interviews (Internal & Customers) • Design an architecture up front • Peer review and sanity checking • Implement by Q2 2013 Tuesday, May 14, 2013
39. The Interviews • 1-on-1 interviews with everyone in the company • Talked with three classes of customers • Ended up with 20+ pages of notes and requirements • #1 requested feature: “OLAP” (UGH) Tuesday, May 14, 2013
40. Caches Aggregator PubNub Client Tuesday, May 14, 2013
41. Caches Aggregator PubNub Client DB MySQL Tuesday, May 14, 2013
42. Caches Aggregator PubNub Client DB MySQL Stats API Tuesday, May 14, 2013
43. API • Built using Sinatra • Indepedent from the core API (via Fastly routing magic) • Highly optimized MySQL Queries (templates using Mustache) • Coverage of most user & internal needs (no OLAP, le sigh) Tuesday, May 14, 2013
44. API • Easy to use time parameters from=2 weeks ago& to=now& by=hour via Chronic (github.com/mojombo/chronic) • Good amount of query types: By Time, Aggregate, Percentile, Usage, etc. Tuesday, May 14, 2013
45. The Interface • Uses d3 • It’s neat-o but not perfect • Displays the information users need the most Tuesday, May 14, 2013
46. DEMO TIME Tuesday, May 14, 2013
47. The Future • “Done” is a myth • Data evolves, so does visualization and reporting • Working on “By Second” historical stats (Using Amazon RedShift) • And a better interface... Tuesday, May 14, 2013
48. FUTURE DEMO TIME Tuesday, May 14, 2013
49. Questions? • Visit https://www.fastly.com • Use fastly ;-) • Follow me on Twitter: @rsandor Tuesday, May 14, 2013