Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Behavioral Databases
Search
benbjohnson
October 24, 2012
Technology
16
2.5k
Behavioral Databases
A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.
benbjohnson
October 24, 2012
Tweet
Share
More Decks by benbjohnson
See All by benbjohnson
Why You Shouldn't Write a Database
benbjohnson
2
560
Static Code Analysis Using SSA
benbjohnson
6
1.8k
Writing a High Performance Database in Go
benbjohnson
11
6.8k
Raft: The Understandable Distributed Consensus Protocol
benbjohnson
105
28k
Behavioral Analytics - Understanding the "why" and "how" of your users.
benbjohnson
9
2k
New Tech - Sky
benbjohnson
2
380
Sky Analytics Pitch Deck
benbjohnson
0
610
Other Decks in Technology
See All in Technology
Introduction to Sansan for Engineers / エンジニア向け会社紹介
sansan33
PRO
5
43k
Performance Insights 廃止から Database Insights 利用へ/transition-from-performance-insights-to-database-insights
emiki
0
320
Databricks AI/BI Genie の「値ディクショナリー」をAmazonの奥地(S3)まで見に行く
kameitomohiro
1
320
ニッポンの人に知ってもらいたいGISスポット
sakaik
0
180
dbtとBigQuery MLで実現する リクルートの営業支援基盤のモデル開発と保守運用
recruitengineers
PRO
3
140
衛星画像超解像化によって実現する2D, 3D空間情報の即時生成と“AI as a Service”/ Real-time generation spatial data enabled_by satellite image super-resolution
lehupa
0
190
Zephyr(RTOS)にEdge AIを組み込んでみた話
iotengineer22
0
210
OpenTelemetry が拡げる Gemini CLI の可観測性
phaya72
2
680
現場データから見える、開発生産性の変化コード生成AI導入・運用のリアル〜 / Changes in Development Productivity and Operational Challenges Following the Introduction of Code Generation AI
nttcom
0
370
Wasmの気になる最新情報
askua
0
160
AIとともに歩んでいくデザイナーの役割の変化
lycorptech_jp
PRO
0
570
AI Agent Dojo #2 watsonx Orchestrateフローの作成
oniak3ibm
PRO
0
130
Featured
See All Featured
Stop Working from a Prison Cell
hatefulcrawdad
271
21k
How to train your dragon (web standard)
notwaldorf
97
6.3k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.9k
Fantastic passwords and where to find them - at NoRuKo
philnash
52
3.4k
Imperfection Machines: The Place of Print at Facebook
scottboms
269
13k
Documentation Writing (for coders)
carmenintech
75
5.1k
Rebuilding a faster, lazier Slack
samanthasiow
84
9.2k
GitHub's CSS Performance
jonrohan
1032
470k
Statistics for Hackers
jakevdp
799
220k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
Large-scale JavaScript Application Architecture
addyosmani
514
110k
Transcript
Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson
My Background
Former Oracle DBA
Former Oracle DBA Behavioral Analytics
Former Oracle DBA Data Visualization Behavioral Analytics
Why did I write a database?
Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K
QPS/core
Processing Needed to Occur Near Data Client Server 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data Client Server 1ms 1ms
1ms 1ms 1ms 1ms 1ms 1ms
Processing Needed to Occur Near Data
Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB
http://www.jcmit.com/memoryprice.htm
Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB
2010 $20/GB http://www.jcmit.com/memoryprice.htm
Traditional Databases Have a Lot Of Features I Don’t Need
Locks & Latches
Transactions
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
Traditional Databases are Limited to Simple Data Access Key/Value Tabular
(Boring)
Traditional Databases Are Not Real-Time
Basics of Behavioral Data
Actions
Actions & State
Actions & State + Time
Clickstream Logs Sensor Data Financial Transactions IVR
Important Differences In Behavioral Data
Behavioral Data is Historical
Behavioral Data is Isolated
Isolated = Concurrency
Performance / Internals
Aggregates 100M events / sec / core
Optimizations Stored by object then time Memory mapped Easy, compact
data format
Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,
No Dependencies
Writing Your Own Language (Tangent)
EQL (Event Query Language)
EQL (Event Query Language)
Qip
Qip (Doesn’t stand for anything)
What is Qip? * LLVM-backed query processing language. * JIT
compiled on the fly. * As fast as C. * Removed from build because of complexity.
GitHub Archive Visualizer (Demo)
Ruby to Sky Integration
Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,
data: { name: “John”, age: 20 } ) )
Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )
What’s Next For Sky?
More Analytics Functions!
Cohort Analysis 1 2 3 4 5 Jan 80% 70%
65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up
DAGs Home Page Sign Up Checkout View Product Cancel Order
Awesome Open Source Analytics Tools It’s like MixPanel that you
can install!
Modules!
Predictive Behavioral Analytics What will your users do next?
Risk Analysis
Anomaly Dectection / Fraud Detection
Questions?
Contact Info @benbjohnson
[email protected]