Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Behavioral Databases
Search
benbjohnson
October 24, 2012
Technology
16
2.4k
Behavioral Databases
A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.
benbjohnson
October 24, 2012
Tweet
Share
More Decks by benbjohnson
See All by benbjohnson
Why You Shouldn't Write a Database
benbjohnson
2
530
Static Code Analysis Using SSA
benbjohnson
6
1.6k
Writing a High Performance Database in Go
benbjohnson
11
6.6k
Raft: The Understandable Distributed Consensus Protocol
benbjohnson
106
27k
Behavioral Analytics - Understanding the "why" and "how" of your users.
benbjohnson
9
1.9k
New Tech - Sky
benbjohnson
2
350
Sky Analytics Pitch Deck
benbjohnson
0
550
Other Decks in Technology
See All in Technology
RAGのサービスをリリースして1年3ヶ月が経ちました
segavvy
4
960
楽しくGoを学び合う、LayerXの勉強会文化 / LayerX's study culture of having fun and learning Go together
ar_tama
2
350
Azure OpenAI Service Dev Day / LLMでできる!使える!生成AIエージェント
masahiro_nishimi
3
810
クラウド利用者の「責任」をどう果たす?AWSセキュリティ対策のススメ #AWSSummit
hiashisan
0
280
How to Think Like a Performance Engineer
csswizardry
4
590
CTOから見た事業開発とプロダクト開発 / My Perspective on Business and Product Development as CTO
keisuke69
4
960
累計ダウンロード数1億8000万を超えるアプリケーションプラットフォームのレガシーシステム脱却とモダン化への道
kmitsuhashi
0
120
Azure AI ことはじめ
tsubakimoto_s
0
130
dxd2024-生成AIに振り回された3か月間の成功と失敗/dxd2024-link-and-motivation
lmi
2
260
Flutter研修【MIXI 24新卒技術研修】
mixi_engineers
PRO
0
160
初中級者用如何使用backlog -VALE TUDOEDITION-
in0u
0
140
データ分析を支える技術 生成AI再入門
ishikawa_satoru
0
380
Featured
See All Featured
Infographics Made Easy
chrislema
238
18k
Debugging Ruby Performance
tmm1
71
11k
For a Future-Friendly Web
brad_frost
173
9.2k
Build The Right Thing And Hit Your Dates
maggiecrowley
28
2.2k
Building Adaptive Systems
keathley
34
2k
Mobile First: as difficult as doing things right
swwweet
219
8.8k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
325
21k
Web Components: a chance to create the future
zenorocha
307
41k
Git: the NoSQL Database
bkeepers
PRO
423
64k
How GitHub (no longer) Works
holman
305
140k
Building Effective Engineering Teams - LeadDev
addyosmani
47
2.2k
Optimising Largest Contentful Paint
csswizardry
18
2.6k
Transcript
Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson
My Background
Former Oracle DBA
Former Oracle DBA Behavioral Analytics
Former Oracle DBA Data Visualization Behavioral Analytics
Why did I write a database?
Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K
QPS/core
Processing Needed to Occur Near Data Client Server 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data
Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB
http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB
2010 $20/GB http://www.jcmit.com/memoryprice.htm
Traditional Databases Have a Lot Of Features I Don’t Need
Locks & Latches
Transactions
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
(Boring)
Traditional Databases Are Not Real-Time
Basics of Behavioral Data
Actions
Actions & State
Actions & State + Time
Clickstream Logs Sensor Data Financial Transactions IVR
Important Differences In Behavioral Data
Behavioral Data is Historical
Behavioral Data is Isolated
Isolated = Concurrency
Performance / Internals
Aggregates 100M events / sec / core
Optimizations Stored by object then time Memory mapped Easy, compact
data format
Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,
No Dependencies
Writing Your Own Language (Tangent)
EQL (Event Query Language)
EQL (Event Query Language)
Qip
Qip (Doesn’t stand for anything)
What is Qip? * LLVM-backed query processing language. * JIT
compiled on the fly. * As fast as C. * Removed from build because of complexity.
GitHub Archive Visualizer (Demo)
Ruby to Sky Integration
Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,
data: { name: “John”, age: 20 } ) )
Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )
What’s Next For Sky?
More Analytics Functions!
Cohort Analysis 1 2 3 4 5 Jan 80% 70%
65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up
DAGs Home Page Sign Up Checkout View Product Cancel Order
Awesome Open Source Analytics Tools It’s like MixPanel that you
can install!
Modules!
Predictive Behavioral Analytics What will your users do next?
Risk Analysis
Anomaly Dectection / Fraud Detection
Questions?
Contact Info @benbjohnson
[email protected]