Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Behavioral Databases
Search
benbjohnson
October 24, 2012
Technology
16
2.4k
Behavioral Databases
A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.
benbjohnson
October 24, 2012
Tweet
Share
More Decks by benbjohnson
See All by benbjohnson
Why You Shouldn't Write a Database
benbjohnson
2
530
Static Code Analysis Using SSA
benbjohnson
6
1.6k
Writing a High Performance Database in Go
benbjohnson
11
6.6k
Raft: The Understandable Distributed Consensus Protocol
benbjohnson
106
27k
Behavioral Analytics - Understanding the "why" and "how" of your users.
benbjohnson
9
1.9k
New Tech - Sky
benbjohnson
2
360
Sky Analytics Pitch Deck
benbjohnson
0
560
Other Decks in Technology
See All in Technology
React Aria で実現する次世代のアクセシビリティ
ryo_manba
4
1.2k
OR学会2024秋_短期収益と将来のオフ方策評価性能を考慮したクーポン割当方策混合比の決定
recruitengineers
PRO
4
450
Google CloudのLLM活用の選択肢を広げるVertex AIのパートナーモデル
nayuts
0
130
やってやろうじゃないかメカアジャイル! / Let's do it, mechanical agile!
psj59129
1
590
テスト”ケース”駆動開発 で手戻りをなくそう
ryohma0510
0
250
PdMはどのように全てのスピードを上げられるか ~ 非連続進化のための具体的な取り組み ~
sansantech
PRO
4
1.2k
App Router を実プロダクトで採用して見えてきた勘所をちょっとだけ紹介
marokanatani
1
920
AI でアップデートする既存テクノロジーと、クラウドエンジニアの生きる道
soracom
PRO
2
550
再考 アクターモデル/ reconsider actor model
ytake
0
270
Analytics-Backed App Widget Development - Served with Jetpack Glance
miyabigouji
0
540
四国クラウドお遍路 2024 in 高知 オープニング
yukataoka
0
200
Tricentisにおけるテスト自動化へのAI活用ご紹介/20240910Shunsuke Katakura
shift_evolve
0
180
Featured
See All Featured
Build your cross-platform service in a week with App Engine
jlugia
228
18k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
227
52k
Music & Morning Musume
bryan
46
6k
Documentation Writing (for coders)
carmenintech
65
4.3k
Fontdeck: Realign not Redesign
paulrobertlloyd
80
5.1k
Web Components: a chance to create the future
zenorocha
308
42k
RailsConf 2023
tenderlove
28
810
The Illustrated Children's Guide to Kubernetes
chrisshort
47
48k
WebSockets: Embracing the real-time Web
robhawkes
59
7.3k
Making Projects Easy
brettharned
113
5.8k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
225
22k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
8.9k
Transcript
Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson
My Background
Former Oracle DBA
Former Oracle DBA Behavioral Analytics
Former Oracle DBA Data Visualization Behavioral Analytics
Why did I write a database?
Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K
QPS/core
Processing Needed to Occur Near Data Client Server 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data
Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB
http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB
2010 $20/GB http://www.jcmit.com/memoryprice.htm
Traditional Databases Have a Lot Of Features I Don’t Need
Locks & Latches
Transactions
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
(Boring)
Traditional Databases Are Not Real-Time
Basics of Behavioral Data
Actions
Actions & State
Actions & State + Time
Clickstream Logs Sensor Data Financial Transactions IVR
Important Differences In Behavioral Data
Behavioral Data is Historical
Behavioral Data is Isolated
Isolated = Concurrency
Performance / Internals
Aggregates 100M events / sec / core
Optimizations Stored by object then time Memory mapped Easy, compact
data format
Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,
No Dependencies
Writing Your Own Language (Tangent)
EQL (Event Query Language)
EQL (Event Query Language)
Qip
Qip (Doesn’t stand for anything)
What is Qip? * LLVM-backed query processing language. * JIT
compiled on the fly. * As fast as C. * Removed from build because of complexity.
GitHub Archive Visualizer (Demo)
Ruby to Sky Integration
Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,
data: { name: “John”, age: 20 } ) )
Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )
What’s Next For Sky?
More Analytics Functions!
Cohort Analysis 1 2 3 4 5 Jan 80% 70%
65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up
DAGs Home Page Sign Up Checkout View Product Cancel Order
Awesome Open Source Analytics Tools It’s like MixPanel that you
can install!
Modules!
Predictive Behavioral Analytics What will your users do next?
Risk Analysis
Anomaly Dectection / Fraud Detection
Questions?
Contact Info @benbjohnson
[email protected]