Large-scale OLAP with Kobayashi

A talk about a large-scale time-series OLAP engine I built on top of Riak called Kobayashi. Watch the talk here https://vimeo.com/42902962

Transcript

1. Large-scale OLAP with Kobayashi Boundary Tech Talks Fri, May 18, 2012 Dietrich Featherston, Boundary @d2fn Friday, October 12, 12
2. Monitoring is an analytics problem Friday, October 12, 12
3. Historical Perspective Friday, October 12, 12
4. 1 minute collection intervals Arbitrary OLAP Friday, October 12, 12
5. Cassandra bitset indexes per dimension query-time sampling Friday, October 12, 12
6. Friday, October 12, 12
7. riak_core + fastbit Friday, October 12, 12
8. apply intelligence to the problem Friday, October 12, 12
9. Arbitrary OLAP requires 2n data cubes where n is dimensionality Friday, October 12, 12
10. dimensions (11) epoch seconds epoch minutes epoch hours meter id source ip source port dest ip dest port interface country network measurements (4) egress packets egress octets ingress packets ingress octets Friday, October 12, 12
11. Total Volume. by Host Port/Protocol Country Network + meter For each aggregation period Friday, October 12, 12
12. Friday, October 12, 12
13. 15 < 211 Friday, October 12, 12
14. 24 hours 2 Months ~10 years 86,400 Observations (per monitored host per query) Friday, October 12, 12
15. 86,400*15 ≈ 1.3M Observations (per monitored host ) Friday, October 12, 12
16. Total Observations (for half a million meters) Friday, October 12, 12
17. { { 100 meters 10 seconds Riak Key Layout < 80KB Friday, October 12, 12
18. Total Observations Friday, October 12, 12
19. Friday, October 12, 12
20. Friday, October 12, 12
21. Friday, October 12, 12
22. Friday, October 12, 12
23. Friday, October 12, 12
24. Bitcask would have been nice LevelDB backend Use leveldb cache to bound memory Friday, October 12, 12
25. Compute your keys Use secondary indexes sparingly Friday, October 12, 12
26. Friday, October 12, 12
27. How do I query the database? Friday, October 12, 12
28. Find 45 minutes of total traffic seen on meters 1, 2, 226, & 301 starting 18 hours ago broken down by traffic type Friday, October 12, 12
29. < 80KB { { 100 meters 10 seconds Atomic Unit of Storage Friday, October 12, 12
30. 0 (0,99) 100 (100,199) 200 (200,299) 300 (300,399) 400 (400,499) t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t18 t19 Time Meter Id Step 1: fetch appropriate blocks (riak) 45 min 1 2 226 301 Friday, October 12, 12
31. 0 (0,99) 100 (100,199) 200 (200,299) 300 (300,399) 400 (400,499) t0 t1 t2 t3 t4 t5 t6 t7 t8 t9 t10 t11 t12 t13 t14 t15 t18 t19 Time Meter Id Step 2: filter 45 min 1 2 226 301 Friday, October 12, 12
32. topk( , 10) Step 3: aggregate and perform top-k 1 2 226 301 45 min + { epochMillis: 1337230140000 portProtocol: "4740:6" ingressPackets: 370482 ingressOctets: 3113782199 egressPackets: 343780 egressOctets: 37126033 }, { epochMillis: 1337230140000 portProtocol: "9092:6" ingressPackets: 440915 ingressOctets: 1816615857 egressPackets: 481237 egressOctets: 1312198133 }, ... Friday, October 12, 12
33. http://computers-r-terrible/ volume_1m_meter_port_protocol/ data? from=-18h& duration=45m parts=1,2,226,301& aggregations=observationDomainId In URL Form Friday, October 12, 12
34. http://computers-r-terrible/ volume_1m_meter_port_protocol/ data? from=-18h& duration=45m& parts=1,2,226,301& aggregations= observationDomainId, epochMillis Arbitrary Aggregations Friday, October 12, 12
35. “unfortunately the project has been blocked for weeks choosing a name” Friday, October 12, 12
36. V Vʹ′ V ≃ Vʹ′ Friday, October 12, 12
37. V Vʹ′ V ≃ Vʹ′ Friday, October 12, 12
38. V Vʹ′ V ≃ Vʹ′ Friday, October 12, 12
39. Future --> Friday, October 12, 12
40. Send expired data to cold storage output in arbitrary time resolution Friday, October 12, 12
41. Open source the data cubing and predicate matching code Query grammar for kobayashi Friday, October 12, 12
42. questions? Friday, October 12, 12