Future of InfluxDB: API and Clustering

Talk I gave at the London Go Meetup

Transcript

1. The future of InfluxDB: Clustering & API Paul Dix CEO & cofounder of InfluxDB paul@influxdb.com @pauldix
3. How many of you have looked at InfluxDB?
4. What it’s for…
5. Metrics
6. Time Series
7. Analytics
8. Events
9. Use Cases
10. DevOps
11. Real-time analytics (user & business)
12. Sensor Data
13. End Intro
14. Influx API v1
15. SQL-ish
16. A query language is an API
17. select percentile(90, value) from response_times group by time(10m) where time > now() - 1d
18. select percentile(90, value) from response_times group by time(10m) where time > now() - 1d Aggregate functions require a group by time
19. What about transformations? Derivative, moving average, sampling
20. select derivative(value) from redis_keys_count where time > now() - 1d
21. select derivative(value) from redis_keys where time > now() - 1d This function doesn’t require group by. It’s not an aggregate
22. Or where clauses on results?
23. select percentile(90, value) from response_times group by time(10m) having percentile > 1000 where time > now() - 1d
24. Filling in missing measurements?
25. Doesn’t feel right
26. Influx API v2
27. Totally unfinished ideas. ! We need your feedback!
28. Databases {! "name": "paulDB",! "retentionPolicies": [! "..."! ],! "defaultRetentionPolicy": "1_week"! }!
29. Retention Policies [! {! "name": "1_week",! "duration": "7d",! "replicationFactor": 1! },! {! "name": "6_months",! "duration": "182d",! "replicationFactor": 3! },! {! "name": "2_years",! "duration": "730d",! "replicationFactor": 3! }! ]!
30. Data Structure Top level series name [! {! "name": "cpu_load",! "values" : [! {! "double": 89.0,! "tags": [“dataCenter/USWest/host/serverA”],! "time": 1412689241000! }! ]! }! ]!
31. Data Structure [! {! "name": "cpu_load",! "values" : [! {! "double": 89.0,! "tags": [“dataCenter/USWest/host/serverA”],! "time": 1412689241000! }! ]! }! ]! Built in column Built in column
32. Tags are hierarchical [! {! "name": "cpu_load",! "values" : [! {! "double": 89.0,! "tags": [“dataCenter/USWest/host/serverA”],! "time": 1412689241000! }! ]! }! ]!
33. Write multiple series [! {! "values" : [! {! "name": "cpu_load",! "double": 89.0,! },! {! "name": "cpu_wait",! "double": 5! }! ],! "time": 1412689241000,! "tags": [“dataCenter/USWest/host/serverA”],! }! ]! Pull common values outside
34. More closely mirrors the sensor and DevOps use case
35. Sequence number optional [! {! "name": "events",! "values" : [! {! "bool": true,! "tags": [“type/click/userId/1”],! }! ],! "setSequenceNumber": true! }! ]! Server assigns
36. Can still have irregular events that occur at the same time in the same series
37. Data types Double, bool, string, bytes
38. Series now only have one value and tags
39. Tags are indexed!
40. Get defined tags select tags(cpu_load)! [! {! "name": "cpu_load",! "columns": ["tag"],! "values": [! ["host"],! ["region"]! ]! }! ]! Query Result
41. Get defined tags by time [! {! "name": "cpu_load",! "columns": ["tag"],! "values": [! ["host"],! ["region"]! ]! }! ]! Query Result select tags(cpu_load)! where time > now() - 1h!
42. Get tags for multiple series [! {! "name": "cpu_load",! "columns": ["tag"],! "values": [! ["host"],! ["region"]! ]! },! {! "name": "cpu_wait",! "columns": ["tag"],! "values": [! ["host"],! ["region"]! ]! }! ]! Query Result select tags(cpu_load), tags(cpu_wait)!
43. Get tag values [! {! "name": "cpu_load",! "columns": ["host"],! "values": [! ["serverA"],! ]! }! ]! Query Result select tag_values(cpu_load, host)!
44. Get compound tag values [! {! "name": "cpu_load",! "columns": [“region","host"],! "values": [! [“us-east“,"serverA"],! [“us-east“,"serverB"],! [“us-west“,"serverC"]! ]! }! ]! Query Result select tag_values(cpu_load, region, host)!
45. Filter by time and tag [! {! "name": "cpu_load",! "columns": [“region","host"],! "values": [! [“us-west“,"serverC"]! ]! }! ]! Query Result select tag_values(cpu_load, host)! where time > now() - 1h! and region = 'USWest'!
46. How many unique series [! {! "name": "cpu_load",! "columns": ["count"],! "values": [! [10157]! ]! }! ]! Query Result select count(tag_values(tags(cpu_load)))!
47. How many unique series [! {! "name": "cpu_load",! "columns": ["count"],! "values": [! [2241]! ]! }! ]! Query Result select count(tag_values(tags(cpu_load)))! where time > now() - 1h! and region = 'USWest'!
48. Queries always scoped by retention! select tags(“6_months”.”cpu_load”)! [! {! "name": "cpu_load",! "columns": ["tag"],! "values": [! ["host"],! ["region"]! ]! }! ]! Query Result
49. Database default {! "name": "paulDB",! "retentionPolicies": [! "..."! ],! "defaultRetentionPolicy": "1_week"! }!
50. Example Queries
51. List names list names! ! -- or see the names for a given retention policy! list names for 6_month!
52. Query raw data select cpu_load! where data_center = 'us-west' ! and host = 'serverA' ! and time > now() - 1h! Query Result Always! default! column [! {! "name": "cpu_load",! "columns": ["double", “time"],! "values": [! [34.2, 1412805662],! ...! ]! }! ]!
53. Query raw data select cpu_load! where data_center = 'us-west' ! and host = 'serverA' ! and time > now() - 1h! Query Result Always! time [! {! "name": "cpu_load",! "columns": ["double", “time"],! "values": [! [34.2, 1412805662],! ...! ]! }! ]!
54. Query raw data Query Result select log_lines.string! where time > now() - 10m! [! {! "name": "log_lines",! "columns": ["string", "time"],! "values": [! ["INFO: stuff here", 1412805662],! ...! ]! }! ]!
55. Query raw data from other retention policy select 6_month.cpu_load! where data_center = 'us-west' ! and host = 'serverA' ! and time > now() - 7d! [! {! "name": "cpu_load",! "columns": [“double”, “time”],! "values": [! [34.2, 1412805600],! ...! ]! }! ]! Query Result
56. Query raw data Query Result select log_lines.string! where time > now() - 10m! [! {! "name": "log_lines",! "columns": [“string”, “time”],! "values": [! ["INFO: stuff here”, 1412805600],! ...! ]! }! ]!
57. Down sample on the fly Query Result select mean(cpu_load)! where data_center = 'us-west'! and host = 'serverA'! and time > now() - 24h! group by time(10m)! [! {! "name": "cpu_load",! "columns": ["double", "time"],! "values": [! [21.1, 1412805662]! ]! }! ]!
58. Down sample on the fly (merge all hosts) Query Result select mean(cpu_load)! where data_center = 'us-west'! and time > now() - 24h! group by time(10m)! [! {! "name": "cpu_load",! "columns": ["double", "time"],! "values": [! [21.1, 1412805662]! ]! }! ]!
59. Down sample on the fly (expanding into many series) Query select mean(cpu_load)! where data_center = 'us-west'! and time > now() - 24h! expand by time(10m), host!
60. Result [! {! "name": "cpu_load",! "tags": {! "data_center": "us-west"! "host": "serverA"! },! "columns": ["double", "time"],! "values": [! [21.1, 1412805662]! ]! },! {! "name": "cpu_load",! "tags": {! "data_center": "us-west"! "host": "serverB"! },! "columns": ["double", "time"],! "values": [! [21.1, 1412805662]! ]! }! ]!
61. Down sample on the fly Query select mean(cpu_load)! where data_center = ‘us-west'! and host in [“serverA”, “serverB”]! and time > now() - 24h! expand by time(10m), host!
62. Down sample from multiple series at the same time Query select mean(cpu_load), max(cpu_wait)! where data_center = ‘us-west'! and host = 'serverA' and! time > now() - 24h! group by time(10m)!
63. Down sample from multiple series at the same time Result [! {! "name": "cpu_load",! "columns": ["mean", "time"],! "values": [...]! },! {! "name": "cpu_wait",! "columns": ["max", "time"],! "values": [...]! }! ]!
64. Query by regex select log_lines.string! where string =~ /error/i! and time > now() - 4h!
65. Query by regex against tag select log_lines.string! where application =~ /^ruby.*/! and time > now() - 1h!
66. Get the top 10 hosts select mean(cpu_load)! where data_center = 'us-west'! and time > now() - 30m! order by double desc! limit 10!
67. These ideas are already outdated…
68. Pulling back to concepts
69. Selecting series
70. filtering
71. transforming
72. merging
73. splitting
74. functional approach seems better
75. like jQuery chaining
76. split("cpu_load", on:"host", where: "dataCenter"="USWest") .percentile(90, window: "10m") .filter(>95) .limit(1) .merge() .limit(10)
77. Just ideas
78. Clustering
79. Two Parts
80. Broker
81. Data Node
82. How writes work
83. Any server Write
84. Broker Broker Broker Any server Write Streaming Raft Cluster
85. Writes are CP
86. Broker Data Node Broker Broker Any server Write
87. Broker Data Node Data Node Broker Broker Any server Write If replication factor = 2
88. Broker Data Node Data Node Broker Broker Any server Write Data Node Data Node Data Node Data Node
89. How Queries Work
90. Data Node Data Node Any server Data Node Data Node Data Node Data Node select mean(cpu_load)! where data_center = 'us-west'! and host = 'serverA'! and time > now() - 24h! group by time(10m)!
91. Data Node Data Node Any server Data Node Data Node Data Node Data Node Compute Locally select mean(cpu_load)! where data_center = 'us-west'! and host = 'serverA'! and time > now() - 24h! group by time(10m)!
92. Data Node Data Node Any server Data Node Data Node Data Node Data Node Send Summary Ticks select mean(cpu_load)! where data_center = 'us-west'! and host = 'serverA'! and time > now() - 24h! group by time(10m)!
93. Clustering Goal: 1-2M values per second
94. Potential Cluster Size: 3-5 Brokers 50 Data Nodes
95. We’re working on these now, feedback welcome!
96. Thanks Paul Dix paul@influxdb.com @pauldix