How to Build a Smart Profiler for Rails

Customers love fast apps. We've got tools to help us figure out how our Rails apps are performing on our development machines, but that's not what matters. It's critical that we also be able to measure how our apps are actually performing in the production environment.

That's the reason we built Skylight, and in this talk, we'll show you how we did it. We'll discuss the underlying principles Skylight uses to profile your apps across multiple production servers, and take a look at our tech stack, including Storm, Kafka, Cassandra, Rails, and Ember.js.

Transcript

3. BOOTSTRAPPED & PROUD
4. $$$
5. , INC. ?
6. What if we could take advantage of our deep knowledge of Rails and pair it with our expertise in building computation systems at scale? You can build something that no one else has done before. ! We didn't just want to build something a little better than our competitors. We wanted to build something that was light years ahead.
7. 3 BREAKTHROUGHS As we were building Skylight, there were three critical breakthroughs that allowed us to deliver an application that was a quantum leap past existing solutions.
8. HONEST RESPONSE TIMES 1
10. “Our average response time for Basecamp right now is 87ms… That sounds fantastic, doesn’t it? And it easily leads you to believe that all is well and that we wouldn’t need to spend any more time optimizing performance.”
11. “Wrong. That average number is completely skewed by tons of super fast responses to feed requests and other cached replies. If you have 1000 requests that return in 5ms, then you can have 200 requests taking 2,000 ms and still get a respectable 170ms average. Useless.”
12. “What we need instead are histograms.”
14. Cache Hits Cache Misses
15. 95th Percentile
16. 95TH PERCENTILE This is the number that matters, because it doesn't capture the average response time. It's more like the average worst response time.
17. 1 USER ≠ 1 REQUEST Users generate many requests. Having one out of every 20 take multiple seconds is an opportunity for them to leave.
18. MARKET OPPORTUNITY
19. ONE PROBLEM
20. IT’S REALLY HARD
21. 1 YEAR OF RESEARCH 6 MONTHS TO REWRITE
22. Your Rails App Agent events
23. ENDPOINT REQUESTS AVERAGE_RESP UsersController#show 2934 203 Calculating average is super easy. That's why everyone does it.
24. ENDPOINT REQUESTS RESP UsersController#show 2934 203 UPDATE response_times SET requests=requests+1, resp=((resp*requests) +new_resp)/requests+1) EACH REQUEST Calculating average is super easy. That's why everyone does it.
25. Your Rails App Agent events
26. Your Rails App Agent Coordinator Your Rails App Agent Your Rails App Agent
27. Low-level control, high-level productivity
28. Coordinator Kafka protobuf Java Kafka is very high throughput.
29. Storm Kafka Spout Bolt Bolt Bolt
30. Storm Spout Bolt Bolt Bolt
31. Storm Spout Bolt Bolt Bolt Cassandra
32. Storm Spout Bolt Bolt Bolt Cassandra
33. CUTTING EDGE ALGORITHMS & DATA STRUCTURES Carl spent nearly a year reading the latest academic literature.
34. Storm Spout Bolt Bolt Bolt Cassandra
35. FOCUS ON ANSWERS NOT DATA 2
37. INSPECTIONS
39. LIGHTNING FAST UI 3 We stream as much data as possible into your browser, so sorting and sifting through data is incredibly fast.
40. FEEDBACK LOOP
44. <div class="month"> {{moment-date date format="MMM"}} </div> <div class="day-of-month"> {{moment-date date format="DD"}} </div>
45. {{calendar-date date=trialEndsAt}}
46. Model Controller Component Component Component
47. BREAKTHROUGHS
48. HONEST RESPONSE TIMES 1 FOCUS ON ANSWERS 2 LIGHTNING FAST UI 3 BREAKTHROUGHS
49. SKYLIGHT IS A SMART PROFILER DATA DETECTIVE CODE SHERPA BUG CATCHER FLAG RAISER FOR RAILS APPS
50. DEMO
51. ONE MORE THING
52. CPU PROFILING Uses Ruby 2.1's new static sampling API so it's really fast. We can sample the stack without doing any allocations, keeping it super lightweight and low- impact on memory
53. AVAILABLE TODAY SKYLIGHT.IO