Behavioral Databases - Speaker Deck

Slide 1

Slide 1 text

Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson

Slide 2

Slide 2 text

My Background

Slide 3

Slide 3 text

Former Oracle DBA

Slide 4

Slide 4 text

Former Oracle DBA Behavioral Analytics

Slide 5

Slide 5 text

Former Oracle DBA Data Visualization Behavioral Analytics

Slide 6

Slide 6 text

Why did I write a database?

Slide 7

Slide 7 text

Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K QPS/core

Slide 8

Slide 8 text

Processing Needed to Occur Near Data Client Server 1ms 1ms

Slide 9

Slide 9 text

Processing Needed to Occur Near Data Client Server 1ms 1ms 1ms 1ms

Slide 10

Slide 10 text

Processing Needed to Occur Near Data Client Server 1ms 1ms 1ms 1ms 1ms 1ms

Slide 11

Slide 11 text

Processing Needed to Occur Near Data Client Server 1ms 1ms 1ms 1ms 1ms 1ms 1ms 1ms

Slide 12

Slide 12 text

Processing Needed to Occur Near Data

Slide 13

Slide 13 text

Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm

Slide 14

Slide 14 text

Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB http://www.jcmit.com/memoryprice.htm

Slide 15

Slide 15 text

Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB 2010 $20/GB http://www.jcmit.com/memoryprice.htm

Slide 16

Slide 16 text

Traditional Databases Have a Lot Of Features I Don’t Need

Slide 17

Slide 17 text

Locks & Latches

Slide 18

Slide 18 text

Transactions

Slide 19

Slide 19 text

Traditional Databases are Limited to Simple Data Access Key/Value Tabular

Slide 20

Slide 20 text

Traditional Databases are Limited to Simple Data Access Key/Value Tabular (Boring)

Slide 21

Slide 21 text

Traditional Databases Are Not Real-Time

Slide 22

Slide 22 text

Basics of Behavioral Data

Slide 23

Slide 23 text

Actions

Slide 24

Slide 24 text

Actions & State

Slide 25

Slide 25 text

Actions & State + Time

Slide 26

Slide 26 text

Clickstream Logs Sensor Data Financial Transactions IVR

Slide 27

Slide 27 text

Important Differences In Behavioral Data

Slide 28

Slide 28 text

Behavioral Data is Historical

Slide 29

Slide 29 text

Behavioral Data is Isolated

Slide 30

Slide 30 text

Isolated = Concurrency

Slide 31

Slide 31 text

Performance / Internals

Slide 32

Slide 32 text

Aggregates 100M events / sec / core

Slide 33

Slide 33 text

Optimizations Stored by object then time Memory mapped Easy, compact data format

Slide 34

Slide 34 text

Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99, No Dependencies

Slide 35

Slide 35 text

Writing Your Own Language (Tangent)

Slide 36

Slide 36 text

EQL (Event Query Language)

Slide 37

Slide 37 text

EQL (Event Query Language)

Slide 38

Slide 38 text

Qip

Slide 39

Slide 39 text

Qip (Doesn’t stand for anything)

Slide 40

Slide 40 text

What is Qip? * LLVM-backed query processing language. * JIT compiled on the ﬂy. * As fast as C. * Removed from build because of complexity.

Slide 41

Slide 41 text

GitHub Archive Visualizer (Demo)

Slide 42

Slide 42 text

Ruby to Sky Integration

Slide 43

Slide 43 text

Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”, data: { name: “John”, age: 20 } ) )

Slide 44

Slide 44 text

Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )

Slide 45

Slide 45 text

What’s Next For Sky?

Slide 46

Slide 46 text

More Analytics Functions!

Slide 47

Slide 47 text

Cohort Analysis 1 2 3 4 5 Jan 80% 70% 65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up