InfluxDB at Paris Data Geeks

Transcript

1. InfluxDB - an open source distributed time series database Paul Dix @pauldix paul@influxdb.com
2. About me…
3. Microsoft, failed startup, Air Force Space Command, McAffee, EastMedia, Mint Digital, KGB (kind of failed startup), failed startup, Benchmark Solutions (failed finance startup), Thomson Reuters, InfluxDB
5. Organizer NYC Machine Learning (4900+ members)
6. Series Editor - “Data & Analytics”
7. Y Combinator (W13)
8. Time series?
9. Metrics
10. Time Series
11. Analytics
12. Events
13. Measurements AND Events Over Time
14. Data model • Databases • Time series (or tables, but you can have millions) • Points (or rows, but column oriented)
15. Data [ { "name": "cpu", "columns": [ "time", "sequence_number", "value", "host" ], "points": [ [1395168540, 1, 56.7, "foo.influxdb.com"], [1395168540, 2, 43.9, "bar.influxdb.com"] ] } ]
16. Everything is indexed by series and time.
17. Simple Install No external dependencies
18. brew update brew install influxdb
19. RPM, Debian packages ! http://influxdb.org/download
20. http://localhost:8083
25. HTTP API Web services built in
26. HTTP API (writes) curl -X POST \ 'http://localhost:8086/db/mydb/series?u=paul&p=pass' \ -d '[{"name":"foo", "columns":["val"], "points": [[3]]}]'
27. Data (with timestamp) [ { "name": "cpu", "columns": ["time", "value", "host"], "points": [ [1395168540, 56.7, "foo.influxdb.com"], [1395168540, 43.9, "bar.influxdb.com"] ] } ]
28. HTTP API (queries) curl 'http://localhost:8086/db/mydb/series?u=paul&p=pass&q=.'
29. SQL-ish select * from events where time > now() - 1h
30. SQL-ish select * from “series with weird chars ()*@#0982#$” where time > now() - 1h
31. Where Regex select line from application_logs where line =~ /.*ERROR.*/ and time > "2014-03-01" and time < "2014-03-03"
32. Only scans the time range Series and time are the primary index
33. Work with many series…
34. Select from Regex select * from /stats\.cpu\..*/ limit 1
35. Downsampling on the fly…
36. Aggregates select percentile(90, value) from response_times group by time(10m) where time > now() - 1d
37. Continuous Downsampling…
38. Continuous queries (summaries) select count(page_id) from events group by time(1h), page_id into events.[page_id]
39. Series per page id select count from events.67 where time > now() - 7d
40. Continuous queries (regex downsampling) select percentile(value, 90) as value from /^stats\.*/ group by time(5m) into percentile.90.5m.:series_name
41. Percentile series per host select value from percentile.90.stats.cpu.host1 where time > now() - 4h
42. Data Collection Client libraries, CollectD, StatsD, Carbon ingestion, OpenTSDB (soon), Riemann (soon)
43. Built-in UI
44. Grafana
45. Behind the scenes
46. #golang
47. Garbage Collector Generational won’t save us
48. MMAP + Unsafe?
49. Storage engines LevelDB, RocksDB, HyperLevelDB, LMDB
50. Range Deletes Wildly Expensive
51. Shards & Shard Spaces
52. Query Parser YACC & Bison
53. Raft Metadata, servers, cluster state
54. Data replication Not write scalable!
55. TCP + Protobuf Intra-cluster communication, queries, replication
56. How data is distributed
57. Shard type Shard struct { Id uint32 StartTime time.Time EndTime time.Time ServerIds []uint32 }
58. Multiple shards per duration Named “split” in the configuration
59. Data for a series for a given interval exists in a shard* *by default, but can be modified
60. hash(database, series) % split
61. Scale out with many series
62. Questions?
63. Thank you! Paul Dix @pauldix paul@influxdb.com