Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Behavioral Databases
Search
benbjohnson
October 24, 2012
Technology
16
2.3k
Behavioral Databases
A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.
benbjohnson
October 24, 2012
Tweet
Share
More Decks by benbjohnson
See All by benbjohnson
Why You Shouldn't Write a Database
benbjohnson
2
520
Static Code Analysis Using SSA
benbjohnson
6
1.5k
Writing a High Performance Database in Go
benbjohnson
11
6.5k
Raft: The Understandable Distributed Consensus Protocol
benbjohnson
106
27k
Behavioral Analytics - Understanding the "why" and "how" of your users.
benbjohnson
9
1.9k
New Tech - Sky
benbjohnson
2
340
Sky Analytics Pitch Deck
benbjohnson
0
540
Other Decks in Technology
See All in Technology
Cracking the KubeCon CfP
inductor
2
250
ワールドカフェI /チューターを改良する / World Café I and Improving the Tutors
ks91
PRO
0
120
非同期推論システムによるコスト削減と信頼性向上
koki_nishihara
0
250
KubeConにproposalを送りたい人へのアドバイス
sat
PRO
3
250
検証を通して見えてきたTiDBの性能特性
lycorptech_jp
PRO
6
3.8k
DevOpsメトリクスとアウトカムの接続にトライ!開発プロセスを通して計測できるメトリクスの活用方法
ham0215
2
240
Azure Container Apps + Bicep 〜 こんな感じで運用しています
kaz29
2
470
ServiceNow Knowledge 24の歩き方 EYストラテジー・アンド・コンサルティング
manarobot
0
200
エンジニアのキャリアをちょっと楽しくする3本の軸/Three Pillars to Make an Engineer's Career More Enjoyable
kwappa
0
2.7k
ここが嬉しいABAC ここが辛いよABAC #再解説+補足編
masahirokawahara
1
270
SIEMを用いて、セキュリティログ分析の可視化と分析を実現し、PDCAサイクルを回してみた
coconala_engineer
0
300
On Your Data を超えていく!
hirotomotaguchi
2
680
Featured
See All Featured
The Cost Of JavaScript in 2023
addyosmani
16
3.9k
For a Future-Friendly Web
brad_frost
172
9k
Why You Should Never Use an ORM
jnunemaker
PRO
51
8.6k
Faster Mobile Websites
deanohume
299
30k
Visualization
eitanlees
136
14k
A better future with KSS
kneath
231
16k
The Cult of Friendly URLs
andyhume
74
5.7k
It's Worth the Effort
3n
180
27k
From Idea to $5000 a Month in 5 Months
shpigford
377
45k
Atom: Resistance is Futile
akmur
259
25k
Music & Morning Musume
bryan
41
5.6k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
6
1.5k
Transcript
Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson
My Background
Former Oracle DBA
Former Oracle DBA Behavioral Analytics
Former Oracle DBA Data Visualization Behavioral Analytics
Why did I write a database?
Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K
QPS/core
Processing Needed to Occur Near Data Client Server 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data
Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB
http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB
2010 $20/GB http://www.jcmit.com/memoryprice.htm
Traditional Databases Have a Lot Of Features I Don’t Need
Locks & Latches
Transactions
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
(Boring)
Traditional Databases Are Not Real-Time
Basics of Behavioral Data
Actions
Actions & State
Actions & State + Time
Clickstream Logs Sensor Data Financial Transactions IVR
Important Differences In Behavioral Data
Behavioral Data is Historical
Behavioral Data is Isolated
Isolated = Concurrency
Performance / Internals
Aggregates 100M events / sec / core
Optimizations Stored by object then time Memory mapped Easy, compact
data format
Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,
No Dependencies
Writing Your Own Language (Tangent)
EQL (Event Query Language)
EQL (Event Query Language)
Qip
Qip (Doesn’t stand for anything)
What is Qip? * LLVM-backed query processing language. * JIT
compiled on the fly. * As fast as C. * Removed from build because of complexity.
GitHub Archive Visualizer (Demo)
Ruby to Sky Integration
Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,
data: { name: “John”, age: 20 } ) )
Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )
What’s Next For Sky?
More Analytics Functions!
Cohort Analysis 1 2 3 4 5 Jan 80% 70%
65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up
DAGs Home Page Sign Up Checkout View Product Cancel Order
Awesome Open Source Analytics Tools It’s like MixPanel that you
can install!
Modules!
Predictive Behavioral Analytics What will your users do next?
Risk Analysis
Anomaly Dectection / Fraud Detection
Questions?
Contact Info @benbjohnson
[email protected]