Refactoring InfluxDB: from Go to Go

Talk given at the Golang SF meetup about the rewrite from 0.8 to 0.9 of InfluxDB

1. Refactoring InfluxDB: from Go to Go Paul Dix CEO and cofounder of InfluxDB @pauldix paul@influxdb.com
2. About me…
4. Organizer NYC Machine Learning (5,500+ members)
5. Series Editor - “Data & Analytics”
6. Cofounder & CEO of InfluxDB
7. Onto the talk!
8. Confession… (not a Go talk)
9. API Design
10. Software Design
11. –InfluxDB design philosophy “Optimize for developer happiness.”
12. Refactor Rewrite
13. Before that, what’s InfluxDB?
15. Open source distributed time series database written in Go with no external dependencies
16. Metrics
17. Time Series
18. Analytics
19. Events
20. Use Cases • Metrics (DevOps) • Sensor Data • Realtime Analytics
21. Features Upcoming 0.9.0 release
22. Data model • Databases
23. Data model • Databases • Measurements • cpu_load, temperature, log_lines, click, etc.
24. Data model • Databases • Measurements • cpu_load, temperature, log_lines, click, etc. • Tags • region=uswest, host=serverA, building=23, service=redis, etc.
25. Data model • Databases • Measurements • cpu_load, temperature, log, click, etc. • Tags • region=uswest, host=serverA, building=23, service=redis, etc. • Series - measurement + unique tagset
26. Data model • Databases • Measurements • cpu_load, temperature, log, click, etc. • Tags • region=uswest, host=serverA, building=23, service=redis, etc. • Series - measurement + unique tagset • Points • Fields - bool, int64, float64, string, []byte • Timestamp - nano epoch
27. Writing Data curl -XPOST 'http://localhost:8086/write' -d '...'!
28. Writing Data {! "database": "mydb",! "retentionPolicy": "30d",! "points": [! {! "name": "cpu_load",! "tags": {! "host": "server01",! "region": "us-west"! },! "timestamp": "2009-11-10T23:00:00Z",! "fields": {! "value": 0.64! }! }! ]! }! Measurement Tags Fields
29. Querying curl -G 'http://localhost:8086/query' --data-urlencode "q=..."!
30. SQL-ish query language
31. SELECT value FROM cpu WHERE host = 'serverA'! {! "results":[! {! "query": "SELECT value FROM cpu WHERE host='serverA'",! "series": [! {! "name": "cpu",! "tags": {! "host": "serverA"! },! "columns": ["time", "value"],! "values": [! ["2009-11-10T23:00:00Z", 22.1],! ["2009-11-10T23:00:10Z", 25.2]! ]! }! ]! }! ]! }! QUERY: RESULTS:
32. SELECT value FROM cpu! WHERE host = ‘serverA'OR host = 'serverB'! QUERY: {! "series": [! {! "name": "cpu",! "tags": {! "host": "serverA"! },! "columns": ["time", "value"],! "values": []! },! {! "name": "cpu",! "tags": {! "host": "serverB"! },! "columns": ["time", "value"],! "values": []! } ! ]! }! SERIES! IN RESULT:
33. SELECT percentile(90, value) FROM cpu! WHERE time > now() - 4h! GROUP BY time(10m), region QUERY: [! {! "name": "cpu",! "tags": {! "region": "us-west"! },! "columns": ["time", "percentile"],! "values": []! },! {! "name": "cpu",! "tags": {! "region": "us-east"! },! "columns": ["time", "percentile"],! "values": []! } ! ]! SERIES! IN RESULT:
34. Multiple aggregates SELECT mean(value), percentile(90, value), min(value), max(value)! FROM cpu! WHERE host='serverA' AND time > now() - 48h! GROUP BY time(1h)!
35. Return every series in CPU SELECT mean(value)! FROM cpu! WHERE time > now() - 48h! GROUP BY time(1h), *!
36. Discovery based on tags
37. {! "results":[! {! "query": "SHOW MEASUREMENTS",! "series": [! {! "name": "measurements",! "columns": ["name"],! "values": [! ["cpu"],! ["memory"],! ["network"]! ]! }! ]! }! ]! }!
38. {! "results":[! {! "query": "SHOW SERIES",! "series": [! {! "name": "cpu",! "columns": ["id", "region", "host"],! "values": [! [1, "us-west", "serverA"],! [2, "us-east", "serverB"]! ]! }! ]! }! ]! }!
39. {! "query": "SHOW MEASUREMENTS WHERE service='redis'",! "series": [! {! "name": "measurements",! "name": "series",! "columns": ["measurement"],! "values": [! ["key_count"],! ["connections"]! ]! }! ]! }!
40. {! "query": "SHOW TAG KEYS from cpu",! "series": [! {! "name": "keys",! "columns": ["key"],! "values": [! ["region"],! ["host"]! ]! }! ]! }!
41. {! "query": "SHOW TAG VALUES WITH KEY = service",! "series": [! {! "name": "series",! "columns": ["service"],! "values": [! ["redis"],! ["apache"]! ]! }! ]! }!
42. {! "query": "SHOW TAG VALUES FROM cpu WITH KEY = service",! "series": [! {! "name": "series",! "columns": ["service"],! "values": [! ["redis"],! ["apache"]! ]! }! ]! }!
43. Much more • Retention policies • Automatic downsampling and aggregation • Clustering
44. onto the rewrite…
45. Since November we’ve been rewriting InfluxDB from scratch.
46. – Joel Spolsky on rewriting from scratch Things You Should Never Do, Part I http://www.joelonsoftware.com/articles/fog0000000069.html “… the single worst strategic mistake that any software company can make …”
47. Why would we rewrite?
48. –InfluxDB design philosophy “Optimize for developer happiness.”
49. What does this mean?
50. for InfluxDB users…
51. simple setup
52. flexible API
53. empowering API
54. performant by default
55. helps developers build analytics applications faster
56. for InfluxDB developers…
57. fast build time
58. idiomatic Go code
59. easy to build and contribute
60. Not Happy
61. Split across three different legitimate areas
62. Feature Requests Moving average, different kinds of derivatives, ways to fill data, top N for a given period, exact data point for min/max
63. Performance Bugs High CPU load on query, where clause takes too long, list series takes too long
64. Bugs open file handles, out of memory, clustering crashes
65. Questions (illegitimate?)
66. Some questions point to bad API design
67. We identified 3 different areas of improvement
68. • API Design • Structural Design Problems Limiting Performance • Underlying Technology Choices Causing Bugs
69. API Design should be understandable, should push users in the right direction
70. Previous API Many series with metadata in the name like in Graphite region.us.data_center.1.host.serverA.network_in! region.us.data_center.1.host.serverA.network_out! 5m.mean.region.us.data_center.1.host.serverA.network_in! 5m.mean.region.us.data_center.1.host.serverA.network_out!
71. Understandable API Design: Retention Policies • Previously called shard spaces • Users tell the server which shard space to read/ write data into based on a regex to match against the series name
72. Too hard to use, couldn’t tell where data was going
73. Users could accidentally hide a bunch of data that was previously available
74. Solution: have the user be explicit at write or query time
75. Pushing users in the right direction: Tags SELECT mean(value) FROM cpu! WHERE host = 'serverA'! Users wanted this:
76. Columns • Wasted space • Add Indexes? • User still has to create them
77. Everyone was expecting tags… Indexed metadata and series
78. Structural Design Problems LIST SERIES /.*region\.uswest.*/! Tell users to have many series: SELECT mean(value)! FROM merge(/.*region\.uswest.*/)! WHERE time > now() - 2h! GROUP BY time(5m)!
79. No way to make that perform well with > 100k series
80. Solution: measurements and tags break up the namespace
81. Structural Design Problems: Query Engine • Pipes raw data over the network • Need to redesign to get data locality • MapReduce framework
82. Solution: refactor/rewrite the query engine
83. Underlying Technology Choices • Protobufs • Performance • Everywhere in code (network, database)
84. Solution: switch to raw bytes for storage
85. Underlying Technology Choices • LevelDB • Too many file handles • No online backups • Too hard to transfer shard from one server to another
86. Solution: switch to BoltDB
87. Underlying Technology Choices • Flex & Bison for Parser • Very hard to understand and update • CGo code
88. Solution: switch to pure Go parser
89. All of these things together pointed to a rewrite Large API changes, underlying technology changes, and code in every area getting touched
90. A refactor would have been a rewrite
91. It would have dragged breaking changes out over multiple releases
92. With a rewrite we rip the bandaid off quickly
93. –InfluxDB design philosophy “Optimize for developer happiness.”
94. The 0.9.0 release gives us a solid foundation to build on
95. Thank you Paul Dix @pauldix paul@influxdb.com ! P.S. we’re hiring Go and front-end developers