Building a real time analytics engine in JRuby

Transcript

1. Building a real time analytics engine in JRuby David Dahl @effata
2. whoami ‣ Senior developer at Burt ‣ Analytics for online advertising ‣ Ruby lovers since 2009 ‣ AWS
6. Getting started ‣ Writing everything to mysql, querying for every report - Broke down on first major campaign ‣ Precalculate all the things! ‣ Every operation in one application - Extremely scary to deploy ‣ Still sticking to MRI
8. Stuck ‣ Separate and buffer with RabbitMQ - Eventmachine ‣ Store stuff with MongoDB - Blocking operations ‣ Bad things
9. Java? ‣ Threading ‣ “Enterprise” ‣ Lots of libraries Think about creating something Java ecosystem Discover someone has made it for you already Profit!
10. Moving to JRuby ‣ Threads! ‣ A real GC ‣ JIT ‣ Every Java, Scala, Ruby lib ever made ‣ Wrapping java libraries is fun! ‣ Bonus: Not hating yourself
11. Challenges
12. “100%” uptime ‣ We can “never” be down! ‣ But we can pause ‣ Don’t want to fail on errors ‣ But it’s ok to die
13. Buffering ‣ Split into isolated services ‣ Add a buffering pipeline between - We LOVE RabbitMQ ‣ Ack and persist in a “transaction” ‣ Figure out if you want - at most once - at least once
14. Databases ‣ Pick the right tool for the job ‣ MongoDB everywhere = bad ‣ Cassandra ‣ Redis ‣ NoDB - keep it streaming!
15. Java.util.concurrent
16. Shortcut
17. Executors Better than doing Thread.new
18. thread_pool = ! Executors.new_fixed_thread_pool(16) stuff.each do |item| thread_pool.submit do crunch_stuff(item) end end
19. Blocking queues Producer/consumer pattern made easy Don’t forget back pressure!
20. queue = ! JavaConcurrent::LinkedBlockingQueue.new # With timeout queue.offer(data, 60, Java::TimeUnit::SECONDS) queue.poll(60, Java::TimeUnit::SECONDS) # Blocking queue.put(data) queue.take
21. Back pressure Storage Timer Data processing Queue State
22. queue = ! JavaConcurrent::ArrayBlockingQueue.new(100) # With timeout queue.offer(data, 60, Java::TimeUnit::SECONDS) queue.poll(60, Java::TimeUnit::SECONDS) # Blocking queue.put(data) queue.take
23. More awesomeness ‣ Java.util.concurrent - Atomic(Boolean/Integer/Long) - ConcurrentHashMap - CountDownLatch / Semaphore ‣ Google Guava ‣ LMAX Disruptor
24. Easy mode ‣ Thread safety is hard ‣ Use j.u.c ‣ Avoid shared mutual state if possible ‣ Back pressure
25. Actors Another layer of abstractions
26. Akka Concurrency library in Scala Most famous for its actor implementation
27. Mikka Small ruby wrapper around Akka
28. class SomeActor < Mikka::Actor def receive(message) # do the thing end end
29. Storm github.com/colinsurprenant/redstorm
30. We broke it But YOU should definitely try it out!
31. Hadoop github.com/iconara/rubydoop
32. module WordCount class Mapper def map(key, value, context) # ... end end class Reducer def reduce(key, value, context) # ... end end end
33. Rubydoop.configure do |input_path, output_path| job 'word_count' do input input_path output output_path mapper WordCount::Mapper reducer WordCount::Reducer output_key Hadoop::Io::Text output_value Hadoop::Io::IntWritable end end
34. Other cool stuff ‣ Hotbunnies ‣ Eurydice ‣ Bundesstrasse ‣ Multimeter
35. Thank you @effata david@burtcorp.com