Behavioral Databases

6c76488dff9b5d9a872dff88f008f88e?s=47 benbjohnson
October 24, 2012
Technology
16
1.8k

Behavioral Databases

A presentation I gave at the Derailed meetup in Denver on October 24th, 2012.

6c76488dff9b5d9a872dff88f008f88e?s=128

benbjohnson

October 24, 2012
Tweet

More Decks by benbjohnson

See All by benbjohnson
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
2
440
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
6
1k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
11
5.3k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
106
25k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
9
1.6k
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
2
290
6c76488dff9b5d9a872dff88f008f88e?s=48 benbjohnson
0
440

Other Decks in Technology

See All in Technology
Bf7fe621f4fe1615c228ef8a79b87282?s=48 shirayanagiryuji
2
680
928b1395afedebb5ee48d44ab917c5f1?s=48 ryusa
1
330
13d936e697fe0f4fa96f926d0a712f6c?s=48 sansanbuildersbox
0
180
1e8c5eaf563f965d3ba48b0054b1d955?s=48 shoichiron
8
3.6k
2102a6b8760bd6f57f672805723dd83a?s=48 line_developers_tw
0
710
F6ceb15fb20b52f49e375cec868a0107?s=48 tokitakumi
4
1.1k
8434e5fcdb11033234455d4b2bfb6bb3?s=48 voyage_tech
0
270
9b600b6d2f592d0944507e989770b9b3?s=48 issei126
1
750
1bfc6e2ed04a895bb36f36b86828b689?s=48 110y
1
180
14493d3489b1441918bfddfe298415d9?s=48 yanaga
0
270
Ea29e2b19d11b48f867ae71d6a6de5df?s=48 opelab
1
250
Dc3f88b2c2c4210e3068d94ce7c25bf2?s=48 terkel
17
4.3k

Featured

See All Featured
2bb923c57a1fdc3a2eb484bb8d565fd2?s=48 ufuk
55
5.2k
9fa239b3154a0169d99ab7bf1422dd12?s=48 marcelosomers
220
14k
C157b16234f1e75e8eac3698c1d4414a?s=48 ddemaree
273
31k
462dad0dd24c568a32dcdb0ae895ae1b?s=48 rasmusluckow
316
18k
91cefdf141106fa10674aae74ce05890?s=48 yeseniaperezcruz
297
31k
Dad095ea7038f89f760419ce475d5d14?s=48 michaelherold
224
8.3k
C7393b7ba7ec9c8890dd77d209fbb3c9?s=48 maltzj
499
36k
48c098b6579220a67a6ba949be6e4b5a?s=48 revolveconf
197
9.5k
Be9caeb9d4ef9944d151af909063ed6e?s=48 samlambert
236
9.8k
1177e050db6bafe62885362edf6e3537?s=48 lauravandoore
8
1.1k
170b760e0147792d0140e59ec78e9893?s=48 eitanlees
108
9.6k
D8fc2580cfaca035f666d9e4ee79a7f7?s=48 mongodb
23
3.7k

Transcript

  1. Behavioral Databases Next Generation NoSQL Analytics By Ben Johnson

  2. My Background

  3. Former Oracle DBA

  4. Former Oracle DBA Behavioral Analytics

  5. Former Oracle DBA Data Visualization Behavioral Analytics

  6. Why did I write a database?

  7. Other Databases Are Too Slow Redis 100K ops/second PostgreSQL 5K

    QPS/core

  8. Processing Needed to Occur Near Data Client Server 1ms 1ms

  9. Processing Needed to Occur Near Data Client Server 1ms 1ms

    1ms 1ms

  10. Processing Needed to Occur Near Data Client Server 1ms 1ms

    1ms 1ms 1ms 1ms

  11. Processing Needed to Occur Near Data Client Server 1ms 1ms

    1ms 1ms 1ms 1ms 1ms 1ms

  12. Processing Needed to Occur Near Data

  13. Memory is getting cheap. Use it. 2010 $20/GB http://www.jcmit.com/memoryprice.htm

  14. Memory is getting cheap. Use it. 2011 $10/GB 2010 $20/GB

    http://www.jcmit.com/memoryprice.htm

  15. Memory is getting cheap. Use it. 2012 $5/GB 2011 $10/GB

    2010 $20/GB http://www.jcmit.com/memoryprice.htm

  16. Traditional Databases Have a Lot Of Features I Don’t Need

  17. Locks & Latches

  18. Transactions

  19. Traditional Databases are Limited to Simple Data Access Key/Value Tabular

  20. Traditional Databases are Limited to Simple Data Access Key/Value Tabular

    (Boring)

  21. Traditional Databases Are Not Real-Time

  22. Basics of Behavioral Data

  23. Actions

  24. Actions & State

  25. Actions & State + Time

  26. Clickstream Logs Sensor Data Financial Transactions IVR

  27. Important Differences In Behavioral Data

  28. Behavioral Data is Historical

  29. Behavioral Data is Isolated

  30. Isolated = Concurrency

  31. Performance / Internals

  32. Aggregates 100M events / sec / core

  33. Optimizations Stored by object then time Memory mapped Easy, compact

    data format

  34. Simplicity Supports Int64, Double, String & Bool MessagePack Encoded C99,

    No Dependencies

  35. Writing Your Own Language (Tangent)

  36. EQL (Event Query Language)

  37. EQL (Event Query Language)

  38. Qip

  39. Qip (Doesn’t stand for anything)

  40. What is Qip? * LLVM-backed query processing language. * JIT

    compiled on the fly. * As fast as C. * Removed from build because of complexity.

  41. GitHub Archive Visualizer (Demo)

  42. Ruby to Sky Integration

  43. Add Event SkyDB.add_event( new Event( object_id:1, timestamp: Time.now, action: “/sign_up”,

    data: { name: “John”, age: 20 } ) )

  44. Next Actions SkyDB.next_actions( [ “/”, “/sign_up”, “/checkout” ] )

  45. What’s Next For Sky?

  46. More Analytics Functions!

  47. Cohort Analysis 1 2 3 4 5 Jan 80% 70%

    65% 63% 62% Feb 83% 73% 70% 69% Mar 87% 78% 75% Apr 89% 80% May 90% Month Signed Up Months After Signing Up

  48. DAGs Home Page Sign Up Checkout View Product Cancel Order

  49. Awesome Open Source Analytics Tools It’s like MixPanel that you

    can install!

  50. Modules!

  51. Predictive Behavioral Analytics What will your users do next?

  52. Risk Analysis

  53. Anomaly Dectection / Fraud Detection

  54. Questions?

  55. Contact Info @benbjohnson benbjohnson@yahoo.com