Behavioral Databases

6c76488dff9b5d9a872dff88f008f88e?s=47 benbjohnson
October 24, 2012
Technology
16
1.8k

Behavioral Databases

A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.

6c76488dff9b5d9a872dff88f008f88e?s=128

benbjohnson

October 24, 2012
Tweet

More Decks by benbjohnson

See All by benbjohnson
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
2
430
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
6
1k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
11
5.2k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
105
25k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
9
1.6k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
2
290
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
0
430

Other Decks in Technology

See All in Technology
Ea9506ab742ba01d1bea53af6f5649b7?s=48 tsuemura
24
13k
6b8e2101579190ad96e747e01c279898?s=48 pauloxnet
0
300
961126d9892ae4df45249529a0721571?s=48 sugamasao
0
310
18f1e3993e9ead63e7a0f70d9eb996bb?s=48 wkm2
0
110
Ca7f3070b0edb9ec3e1b5b5109920a8a?s=48 hiyokotaisa
2
1.5k
Eea9a05e6e222a3d50c73f54a49fadf4?s=48 rtechkouhou
0
480
Cd891a89dfec6ca7bd278b5f5bd87c90?s=48 tzkoba
15
4.5k
4b2f3a64637b51e81813accbe8a98083?s=48 miura55
0
250
D6fc8a819afa94f464fe322032a6687f?s=48 mahito
3
1.1k
53850955f15249a1a9dc49df6113e400?s=48 line_developers
0
130
Ecad9d801d79f6c6e5df93094690685e?s=48 sinsoku
6
840
75fd80a8ace6ab292a7de563e018ae11?s=48 takashi_toyosaki
0
120

Featured

See All Featured
7fa6101cd43067653cf4ac6c7ca54305?s=48 akmur
252
19k
E43f8dfd01057f8304f5f8fb9a294d7d?s=48 jeffersonlam
323
14k
32de0bd2ba869609d26fd052a4622778?s=48 cromwellryan
99
5.7k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
196
19k
9375a9529679f1b42b567a640d775e7d?s=48 schacon
144
6.4k
Fde6c3a50ca518a909ef35c22c0ee0e4?s=48 destraynor
222
46k
39488f9d172ab92fd352f2cd7b73258d?s=48 mza
80
3.9k
7c8469ee8c9e594c65c59b919626c08d?s=48 scottboms
250
11k
A60068bce2e73de3a37ca9d2dbe36092?s=48 bryan
28
3.1k
A9a491b0fcbe0fbce3d64063a37add99?s=48 rmw
7
540
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
12
420
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
8
910

Transcript

  1. Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson

  2. My Background

  3. Former Oracle DBA

  4. Former Oracle DBA Behavioral Analytics

  5. Former Oracle DBA Data Visualization Behavioral Analytics

  6. Why did I write a database?

  7. Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K

    QPS/core

  8. Processing Needed to Occur Near Data Client Server 1ms 1ms

  9. Processing Needed to Occur Near Data Client Server 1ms 1ms

    1ms 1ms

  10. Processing Needed to Occur Near Data Client Server 1ms 1ms

    1ms 1ms 1ms 1ms

  11. Processing Needed to Occur Near Data Client Server 1ms 1ms

    1ms 1ms 1ms 1ms 1ms 1ms

  12. Processing Needed to Occur Near Data

  13. Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm

  14. Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB

    http://www.jcmit.com/memoryprice.htm

  15. Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB

    2010 $20/GB http://www.jcmit.com/memoryprice.htm

  16. Traditional Databases Have a Lot Of Features I Don’t Need

  17. Locks & Latches

  18. Transactions

  19. Traditional Databases are Limited to Simple Data Access Key/Value Tabular

  20. Traditional Databases are Limited to Simple Data Access Key/Value Tabular

    (Boring)

  21. Traditional Databases Are Not Real-Time

  22. Basics of Behavioral Data

  23. Actions

  24. Actions & State

  25. Actions & State + Time

  26. Clickstream Logs Sensor Data Financial Transactions IVR

  27. Important Differences In Behavioral Data

  28. Behavioral Data is Historical

  29. Behavioral Data is Isolated

  30. Isolated = Concurrency

  31. Performance / Internals

  32. Aggregates 100M events / sec / core

  33. Optimizations Stored by object then time Memory mapped Easy, compact

    data format

  34. Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,

    No Dependencies

  35. Writing Your Own Language (Tangent)

  36. EQL (Event Query Language)

  37. EQL (Event Query Language)

  38. Qip

  39. Qip (Doesn’t stand for anything)

  40. What is Qip? * LLVM-backed query processing language. * JIT

    compiled on the fly. * As fast as C. * Removed from build because of complexity.

  41. GitHub Archive Visualizer (Demo)

  42. Ruby to Sky Integration

  43. Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,

    data: { name: “John”, age: 20 } ) )

  44. Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )

  45. What’s Next For Sky?

  46. More Analytics Functions!

  47. Cohort Analysis 1 2 3 4 5 Jan 80% 70%

    65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up

  48. DAGs Home Page Sign Up Checkout View Product Cancel Order

  49. Awesome Open Source Analytics Tools It’s like MixPanel that you

    can install!

  50. Modules!

  51. Predictive Behavioral Analytics What will your users do next?

  52. Risk Analysis

  53. Anomaly Dectection / Fraud Detection

  54. Questions?

  55. Contact Info @benbjohnson benbjohnson@yahoo.com