Building InfluxDB, an open source distributed events database

Slides for the InfluxDB talk I gave on March 25th, 2014 in Charlottesville.

Transcript

1. Building InfluxDB, an open source distributed events database Paul Dix @pauldix paul@influxdb.com
2. About me… • Organizer NYC Machine Learning (4600+ members) • Author “Service Oriented Design with Ruby and Rails” • Creator of Feedjira, Typhoeus, and others • Columbia 2009* (CS) • YC (W13)
3. Series Editor - “Data & Analytics”
4. Our YC Company Platform for real-time metrics and monitoring
5. How was YC? • Awesome • Time to iterate • Fantastic network
6. How we arrived here • Had to build InfluxDB to even get to the product • Crowded space with no clear differentiation for us • Better opportunity as an open source platform
7. Platforms should be open source.
8. Monitorama and the 6 week pivot
9. How was pivoting?
10. At first, it sucked.
11. And then it was awesome!
12. InfluxDB - an events database
13. time series, metrics, discrete events, analytics Events Database?
14. Metrics
15. Time Series
16. Analytics
17. Events
18. What do other people do? • DB (Hbase, Cassandra, MySQL, Redis, etc) • Web services • Cron jobs or long running workers for summary tables
19. Building anything with an analytics component is like writing a web application in 1998.
20. Project Goals • Store metrics AND events • Horizontally scalable • Nothing else to run and manage • Shouldn’t have to write server code for downsampling/summaries
21. InfluxDB • Written in Go • LevelDB for storage engine • No dependencies (other than glibc) • Distributed • HTTP API • SQL like query language
22. Cool Integrations • StatsD Backend • CollectD Proxy • CLI (ruby or node) • Libraries for many languages • Graphite • OpenTSDB (soon)
23. Grafana
24. Data [ { "name": "events", "columns": ["type", "email"], "points": [ ["signup", "paul@influxdb.com"], ["paid", "foo@asdf.com"] ] } ]
25. Data (with timestamp) [ { "name": "cpu", "columns": ["time", "value", "host"], "points": [ [1395168540, 56.7, "foo.influxdb.com"], [1395168540, 43.9, "bar.influxdb.com"] ] } ]
26. Storage • Schema-less • Indexed on (series, column, time, sequence number) • Microsecond precision • Doesn’t store null column values • Millions of series
27. SQL-ish select * from some_series where time > now() - 1h
28. Aggregates select percentile(90, value) from some_series group by time(10m) where time > now() - 1d
29. Select from Regex select * from /stats\.cpu\..*/ limit 1
30. Where against Regex select value from some_log_series where value =~ /.*ERROR.*/ and time > "2014-03-01" and time < "2014-03-03"
31. Continuous queries (fan out) select * from events into events.[user_id]
32. Continuous queries (summaries) select count(page_id) from events group by time(1h), page_id into events.[page_id]
33. Continuous queries (regex downsampling) select max(value), context from /stats\.*/ group by time(5m) into max.:series_name
34. Tooling considerations
35. Why Go?
36. Quicker and easier than C or C++ Libraries, memory managed, previous experience.
37. Existing Raft Implementation (goraft)
38. Compiled binary with no dependencies
39. Good enough performance*
40. Community and language momentum NEW SHINY THINGS!!1
41. Why NOT Go?
42. GC Mark and sweep. Large heaps if we want to cache.
43. GC Lack of control over memory. Even generational is sadness.
44. How data is distributed • Shards - contiguous blocks of time • Replication factor (read scalability) • Split - break blocks of time into multiple shards (write scalability)
45. Split • Hashing db, series • Series data always lives in the same shard • Match regex and randomly distribute • Writes scalability at the cost of losing data locality
46. Now a bit of code… • Shards - contiguous blocks of time • Servers have multiple shards • Query can come into any server and hit multiple shards (local or remote)
47. responses := make([]chan *protocol.Response, 0) // shards are arranged by time. // set up response channels to collect results for _, shard := range shards { responseChan := make(chan *protocol.Response) // parallelize it! go shard.Query(querySpec, responseChan) responses = append(responses, responseChan) }
48. // loop through in order pulling results. // Since shards are ordered they come back // in the right order. for _, responseChan := range responses { for { // ordered, processed results onto response. response := <-responseChan if *response.Type == endStreamResponse { break } } }
49. ! ! // shard.go func (s *Shard) Query(q *QuerySpec, c chan *protocol.Response) { req := s.createRequest(q) s.server.Request(req, c) }
50. ! func (s *Server) Request(r *protocol.Request, c chan *protocol.Response) { // sends request over the wire and returns } ! // single goroutine handles responses from server func (s *Server) handleResponses() { for { r := <-s.responses // returns the chan that was sent with // call to Request c := s.getResponseChannel(r) c <- r } }
51. This caused a race condition that took days to debug…
52. responses := make([]chan *protocol.Response, 0) // shards are arranged by time. // set up response channels to collect results for _, shard := range shards { responseChan := make(chan *protocol.Response) // parallelize it! go shard.Query(querySpec, responseChan) responses = append(responses, responseChan) }
53. Single goroutine to read responses and send to channels
54. responses := make([]chan *protocol.Response, 0) // shards are arranged by time. // set up response channels to collect results for _, shard := range shards { responseChan := make(chan *protocol.Response, 10) // parallelize it! go shard.Query(querySpec, responseChan) responses = append(responses, responseChan) }
55. How big should the buffer be?
56. What if we run out of memory?
57. Channels and Goroutines are awesome, but they’re not magic.
58. Open Questions…
59. Is there a better pattern for distributed queries?
60. Is there a way to get around GC? Memcashier and MMap
61. Exciting features coming up… • Copy/move shards within the cluster • Custom functions in JS • Binary Protocol and Pubsub
62. Our goal is to make it as easy to build an analytics product as it is to write a web application.
63. Thank you! Paul Dix @pauldix paul@influxdb.com