Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Building a real time analytics engine in JRuby
Search
Sponsored
·
SiteGround - Reliable hosting with speed, security, and support you can count on.
→
David Dahl
March 02, 2013
Programming
550
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Building a real time analytics engine in JRuby
David Dahl
March 02, 2013
Transcript
35 slides
Building a real time analytics engine in JRuby David Dahl
@effata
whoami ‣ Senior developer at Burt ‣ Analytics for online
advertising ‣ Ruby lovers since 2009 ‣ AWS
None
None
None
Getting started ‣ Writing everything to mysql, querying for every
report - Broke down on first major campaign ‣ Precalculate all the things! ‣ Every operation in one application - Extremely scary to deploy ‣ Still sticking to MRI
None
Stuck ‣ Separate and buffer with RabbitMQ - Eventmachine ‣
Store stuff with MongoDB - Blocking operations ‣ Bad things
Java? ‣ Threading ‣ “Enterprise” ‣ Lots of libraries Think
about creating something Java ecosystem Discover someone has made it for you already Profit!
Moving to JRuby ‣ Threads! ‣ A real GC ‣
JIT ‣ Every Java, Scala, Ruby lib ever made ‣ Wrapping java libraries is fun! ‣ Bonus: Not hating yourself
Challenges
“100%” uptime ‣ We can “never” be down! ‣ But
we can pause ‣ Don’t want to fail on errors ‣ But it’s ok to die
Buffering ‣ Split into isolated services ‣ Add a buffering
pipeline between - We LOVE RabbitMQ ‣ Ack and persist in a “transaction” ‣ Figure out if you want - at most once - at least once
Databases ‣ Pick the right tool for the job ‣
MongoDB everywhere = bad ‣ Cassandra ‣ Redis ‣ NoDB - keep it streaming!
Java.util.concurrent
Shortcut
Executors Better than doing Thread.new
thread_pool = ! Executors.new_fixed_thread_pool(16) stuff.each do |item| thread_pool.submit do crunch_stuff(item)
end end
Blocking queues Producer/consumer pattern made easy Don’t forget back pressure!
queue = ! JavaConcurrent::LinkedBlockingQueue.new # With timeout queue.offer(data, 60, Java::TimeUnit::SECONDS)
queue.poll(60, Java::TimeUnit::SECONDS) # Blocking queue.put(data) queue.take
Back pressure Storage Timer Data processing Queue State
queue = ! JavaConcurrent::ArrayBlockingQueue.new(100) # With timeout queue.offer(data, 60, Java::TimeUnit::SECONDS)
queue.poll(60, Java::TimeUnit::SECONDS) # Blocking queue.put(data) queue.take
More awesomeness ‣ Java.util.concurrent - Atomic(Boolean/Integer/Long) - ConcurrentHashMap - CountDownLatch
/ Semaphore ‣ Google Guava ‣ LMAX Disruptor
Easy mode ‣ Thread safety is hard ‣ Use j.u.c
‣ Avoid shared mutual state if possible ‣ Back pressure
Actors Another layer of abstractions
Akka Concurrency library in Scala Most famous for its actor
implementation
Mikka Small ruby wrapper around Akka
class SomeActor < Mikka::Actor def receive(message) # do the thing
end end
Storm github.com/colinsurprenant/redstorm
We broke it But YOU should definitely try it out!
Hadoop github.com/iconara/rubydoop
module WordCount class Mapper def map(key, value, context) # ...
end end class Reducer def reduce(key, value, context) # ... end end end
Rubydoop.configure do |input_path, output_path| job 'word_count' do input input_path output
output_path mapper WordCount::Mapper reducer WordCount::Reducer output_key Hadoop::Io::Text output_value Hadoop::Io::IntWritable end end
Other cool stuff ‣ Hotbunnies ‣ Eurydice ‣ Bundesstrasse ‣
Multimeter
Thank you @effata
[email protected]
More Decks by David Dahl
See All by David Dahl
Nosql - getting over the bad parts
effata
1
120
Other Decks in Programming
See All in Programming
例外からRich Errorsまで Kotlinエラーハンドリングの進化を読み解く
shiita0903
0
240
地方こそサーバーレス — 10年続けた、顔が見えるチームで確かな基盤を / Serverless Matters Most in Regional Areas — 10 Years of Building Reliable Foundations with Close-Knit Teams
seike460
PRO
2
510
正解の無い問題を 「できるだけ上手く」解く ヒューリスティックで感性を磨こう。〜現実課題をプログラミングで解きあうコミュニティについて〜
allegrogiken
0
110
React Native에 현대적인 빌드 도구 더하기: Rollipop 구현기
leegeunhyeok
0
190
Growing PicoRuby's Ecosystem on ESP32
yuuu
2
150
Agentic AI with the Power of Jakarta EE
ivargrimstad
0
180
仕様駆動開発入門 (何を作るかを先に決めて渡す) | 目指せ Kiro マスター 05
inariku
PRO
3
220
作れるエンジニアから、価値を届けるエンジニアへ
ryunakayama
1
780
数行の変更でDockerビルドを最大91%高速化~GitLab CIの実行基盤を社内40リポジトリに展開するまで~
picopico
1
200
エージェントの管理もエージェントに – herdr-core と自作バックログで作る司令塔 –
nearme_tech
PRO
1
110
Ruby から Emacs を操作したい
phigasui
0
390
TanStack Vue Query と Pinia Colada から学ぶ非同期状態の型設計
aoshi
0
700
Featured
See All Featured
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
35
2.9k
A better future with KSS
kneath
240
18k
The SEO identity crisis: Don't let AI make you average
varn
0
630
Visual Storytelling: How to be a Superhuman Communicator
reverentgeek
2
750
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
25
2.2k
Balancing Empowerment & Direction
lara
6
1.4k
Making Projects Easy
brettharned
120
6.9k
What the history of the web can teach us about the future of AI
inesmontani
PRO
3
830
Optimizing for Happiness
mojombo
378
72k
Navigating Algorithm Shifts & AI Overviews - #SMXNext
aleyda
1
1.7k
Well, that’s weird…
csswizardry
1
200
職位にかかわらず全員がリーダーシップを発揮するチーム作り / Building a team where everyone can demonstrate leadership regardless of position
madoxten
70
69k