Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Behavioral Databases

6c76488dff9b5d9a872dff88f008f88e?s=47 benbjohnson
October 24, 2012
Technology
16
2k

Behavioral Databases

A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.

6c76488dff9b5d9a872dff88f008f88e?s=128

benbjohnson

October 24, 2012
Tweet

More Decks by benbjohnson

See All by benbjohnson
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
2
470
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
6
1.1k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
11
5.6k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
106
26k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
9
1.7k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
2
300
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
0
450

Other Decks in Technology

See All in Technology
Cd931bc05e7cee46b9a950a63e47ba4c?s=48 you
0
210
F1f3e75b74890a8689878a88873378a9?s=48 inayamafumitaka
0
130
E55fbffb4afd0a84d36c945ed68d8c19?s=48 caddi_eng
0
120
B49933741d74e122bc1314b2975e9fc9?s=48 mrtc0
3
290
97d180294474e9df01503148f3b11f39?s=48 lambda
0
1.5k
1f7f6943810323d2ad8ac736617047d5?s=48 johntitor
0
480
E163ae0588e154424d78e17672076ef4?s=48 kosugitti
3
470
Be2a68a5ffde6683d2a5bd213ce8b814?s=48 aviciida
1
370
Bf7fe621f4fe1615c228ef8a79b87282?s=48 shirayanagiryuji
0
100
A35557d139d8d53246763a01ec847a3f?s=48 kichiemon
2
260
53850955f15249a1a9dc49df6113e400?s=48 line_developers
PRO
4
290
179d41d0fdd38c6980de9f2ae0e4768a?s=48 9wick
0
110

Featured

See All Featured
1519587b1a0febb658c36c94f6272b0d?s=48 roundedbygravity
85
8.1k
3d4519fd34b76fb265fc0237f3792bd4?s=48 danielanewman
6
740
245cee81a9c424266e5e401d844ea881?s=48 lara
175
10k
E4f5d494ebdc9e7cce1aecf3ce3e8bc1?s=48 shpigford
167
19k
168ccec72eee0530b818d44f3fedaacf?s=48 shlominoach
177
8.1k
06f8b41980eb4c577fa40c41d5030c19?s=48 keathley
25
910
E9221b172e78e888355fa2fe4541b595?s=48 mclloyd
PRO
2
7.4k
282d599b414022eba5096510f675b6e2?s=48 chrislema
231
16k
9375a9529679f1b42b567a640d775e7d?s=48 schacon
152
6.8k
245cee81a9c424266e5e401d844ea881?s=48 lara
26
3.4k
F57e2136eb9895eb95ff5dc8a1c02677?s=48 zakiwarfel
91
3.6k
D47656e20ff5e42f125fc5ea0fd636c6?s=48 tmm1
64
10k

Transcript

  1. Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson

  2. My Background

  3. Former Oracle DBA

  4. Former Oracle DBA Behavioral Analytics

  5. Former Oracle DBA Data Visualization Behavioral Analytics

  6. Why did I write a database?

  7. Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K

    QPS/core

  8. Processing Needed to Occur Near Data Client Server 1ms 1ms

  9. Processing Needed to Occur Near Data Client Server 1ms 1ms

    1ms 1ms

  10. Processing Needed to Occur Near Data Client Server 1ms 1ms

    1ms 1ms 1ms 1ms

  11. Processing Needed to Occur Near Data Client Server 1ms 1ms

    1ms 1ms 1ms 1ms 1ms 1ms

  12. Processing Needed to Occur Near Data

  13. Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm

  14. Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB

    http://www.jcmit.com/memoryprice.htm

  15. Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB

    2010 $20/GB http://www.jcmit.com/memoryprice.htm

  16. Traditional Databases Have a Lot Of Features I Don’t Need

  17. Locks & Latches

  18. Transactions

  19. Traditional Databases are Limited to Simple Data Access Key/Value Tabular

  20. Traditional Databases are Limited to Simple Data Access Key/Value Tabular

    (Boring)

  21. Traditional Databases Are Not Real-Time

  22. Basics of Behavioral Data

  23. Actions

  24. Actions & State

  25. Actions & State + Time

  26. Clickstream Logs Sensor Data Financial Transactions IVR

  27. Important Differences In Behavioral Data

  28. Behavioral Data is Historical

  29. Behavioral Data is Isolated

  30. Isolated = Concurrency

  31. Performance / Internals

  32. Aggregates 100M events / sec / core

  33. Optimizations Stored by object then time Memory mapped Easy, compact

    data format

  34. Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,

    No Dependencies

  35. Writing Your Own Language (Tangent)

  36. EQL (Event Query Language)

  37. EQL (Event Query Language)

  38. Qip

  39. Qip (Doesn’t stand for anything)

  40. What is Qip? * LLVM-backed query processing language. * JIT

    compiled on the fly. * As fast as C. * Removed from build because of complexity.

  41. GitHub Archive Visualizer (Demo)

  42. Ruby to Sky Integration

  43. Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,

    data: { name: “John”, age: 20 } ) )

  44. Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )

  45. What’s Next For Sky?

  46. More Analytics Functions!

  47. Cohort Analysis 1 2 3 4 5 Jan 80% 70%

    65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up

  48. DAGs Home Page Sign Up Checkout View Product Cancel Order

  49. Awesome Open Source Analytics Tools It’s like MixPanel that you

    can install!

  50. Modules!

  51. Predictive Behavioral Analytics What will your users do next?

  52. Risk Analysis

  53. Anomaly Dectection / Fraud Detection

  54. Questions?

  55. Contact Info @benbjohnson benbjohnson@yahoo.com