Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Behavioral Databases
Search
benbjohnson
October 24, 2012
Technology
16
2.5k
Behavioral Databases
A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.
benbjohnson
October 24, 2012
Tweet
Share
More Decks by benbjohnson
See All by benbjohnson
Why You Shouldn't Write a Database
benbjohnson
2
550
Static Code Analysis Using SSA
benbjohnson
6
1.7k
Writing a High Performance Database in Go
benbjohnson
11
6.8k
Raft: The Understandable Distributed Consensus Protocol
benbjohnson
106
28k
Behavioral Analytics - Understanding the "why" and "how" of your users.
benbjohnson
9
2k
New Tech - Sky
benbjohnson
2
380
Sky Analytics Pitch Deck
benbjohnson
0
600
Other Decks in Technology
See All in Technology
本当にわかりやすいAIエージェント入門
segavvy
10
5.9k
Jitera Company Deck / JP
jitera
0
140
「手を動かした者だけが世界を変える」ソフトウェア開発だけではない開発者人生
onishi
8
3k
生成AIによる情報システムへのインパクト
taka_aki
1
110
データ駆動経営の道しるべ:プロダクト開発指標の戦略的活用法
ham0215
2
230
公開初日に個人環境で試した Gemini CLI 体験記など / Gemini CLI実験レポート
you
PRO
3
320
An introduction to Claude Code SDK
choplin
3
3.3k
SRE with AI:実践から学ぶ、運用課題解決と未来への展望
yoshiiryo1
1
680
PHPからはじめるコンピュータアーキテクチャ / From Scripts to Silicon: A Journey Through the Layers of Computing
tomzoh
2
380
Data Engineering Study#30 LT資料
tetsuroito
1
560
OTel 公式ドキュメント翻訳 PJ から始めるコミュニティ活動/Community activities starting with the OTel official document translation project
msksgm
0
220
DATA+AI SummitとSnowflake Summit: ユーザから見た共通点と相違点 / DATA+AI Summit and Snowflake Summit
nttcom
0
210
Featured
See All Featured
What’s in a name? Adding method to the madness
productmarketing
PRO
23
3.6k
Why You Should Never Use an ORM
jnunemaker
PRO
58
9.5k
Adopting Sorbet at Scale
ufuk
77
9.5k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.5k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
47
9.6k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
667
120k
Git: the NoSQL Database
bkeepers
PRO
431
65k
Into the Great Unknown - MozCon
thekraken
40
1.9k
Fireside Chat
paigeccino
37
3.5k
Java REST API Framework Comparison - PWX 2021
mraible
31
8.7k
The World Runs on Bad Software
bkeepers
PRO
70
11k
Transcript
Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson
My Background
Former Oracle DBA
Former Oracle DBA Behavioral Analytics
Former Oracle DBA Data Visualization Behavioral Analytics
Why did I write a database?
Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K
QPS/core
Processing Needed to Occur Near Data Client Server 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data
Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB
http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB
2010 $20/GB http://www.jcmit.com/memoryprice.htm
Traditional Databases Have a Lot Of Features I Don’t Need
Locks & Latches
Transactions
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
(Boring)
Traditional Databases Are Not Real-Time
Basics of Behavioral Data
Actions
Actions & State
Actions & State + Time
Clickstream Logs Sensor Data Financial Transactions IVR
Important Differences In Behavioral Data
Behavioral Data is Historical
Behavioral Data is Isolated
Isolated = Concurrency
Performance / Internals
Aggregates 100M events / sec / core
Optimizations Stored by object then time Memory mapped Easy, compact
data format
Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,
No Dependencies
Writing Your Own Language (Tangent)
EQL (Event Query Language)
EQL (Event Query Language)
Qip
Qip (Doesn’t stand for anything)
What is Qip? * LLVM-backed query processing language. * JIT
compiled on the fly. * As fast as C. * Removed from build because of complexity.
GitHub Archive Visualizer (Demo)
Ruby to Sky Integration
Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,
data: { name: “John”, age: 20 } ) )
Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )
What’s Next For Sky?
More Analytics Functions!
Cohort Analysis 1 2 3 4 5 Jan 80% 70%
65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up
DAGs Home Page Sign Up Checkout View Product Cancel Order
Awesome Open Source Analytics Tools It’s like MixPanel that you
can install!
Modules!
Predictive Behavioral Analytics What will your users do next?
Risk Analysis
Anomaly Dectection / Fraud Detection
Questions?
Contact Info @benbjohnson
[email protected]