Refactoring InfluxDB: from Go to Go

39b7a68b6cbc43ec7683ad0bcc4c9570?s=47 Paul Dix
February 19, 2015
6.4k

Refactoring InfluxDB: from Go to Go

Talk given at the Golang SF meetup about the rewrite from 0.8 to 0.9 of InfluxDB

39b7a68b6cbc43ec7683ad0bcc4c9570?s=128

Paul Dix

February 19, 2015
Tweet

Transcript

  1. Refactoring InfluxDB: from Go to Go Paul Dix CEO and

    cofounder of InfluxDB @pauldix paul@influxdb.com
  2. About me…

  3. None
  4. Organizer NYC Machine Learning (5,500+ members)

  5. Series Editor - “Data & Analytics”

  6. Cofounder & CEO of InfluxDB

  7. Onto the talk!

  8. Confession… (not a Go talk)

  9. API Design

  10. Software Design

  11. –InfluxDB design philosophy “Optimize for developer happiness.”

  12. Refactor Rewrite

  13. Before that, what’s InfluxDB?

  14. None
  15. Open source distributed time series database written in Go with

    no external dependencies
  16. Metrics

  17. Time Series

  18. Analytics

  19. Events

  20. Use Cases • Metrics (DevOps) • Sensor Data • Realtime

    Analytics
  21. Features Upcoming 0.9.0 release

  22. Data model • Databases

  23. Data model • Databases • Measurements • cpu_load, temperature, log_lines,

    click, etc.
  24. Data model • Databases • Measurements • cpu_load, temperature, log_lines,

    click, etc. • Tags • region=uswest, host=serverA, building=23, service=redis, etc.
  25. Data model • Databases • Measurements • cpu_load, temperature, log,

    click, etc. • Tags • region=uswest, host=serverA, building=23, service=redis, etc. • Series - measurement + unique tagset
  26. Data model • Databases • Measurements • cpu_load, temperature, log,

    click, etc. • Tags • region=uswest, host=serverA, building=23, service=redis, etc. • Series - measurement + unique tagset • Points • Fields - bool, int64, float64, string, []byte • Timestamp - nano epoch
  27. Writing Data curl -XPOST 'http://localhost:8086/write' -d '...'!

  28. Writing Data {! "database": "mydb",! "retentionPolicy": "30d",! "points": [! {!

    "name": "cpu_load",! "tags": {! "host": "server01",! "region": "us-west"! },! "timestamp": "2009-11-10T23:00:00Z",! "fields": {! "value": 0.64! }! }! ]! }! Measurement Tags Fields
  29. Querying curl -G 'http://localhost:8086/query' --data-urlencode "q=..."!

  30. SQL-ish query language

  31. SELECT value FROM cpu WHERE host = 'serverA'! {! "results":[!

    {! "query": "SELECT value FROM cpu WHERE host='serverA'",! "series": [! {! "name": "cpu",! "tags": {! "host": "serverA"! },! "columns": ["time", "value"],! "values": [! ["2009-11-10T23:00:00Z", 22.1],! ["2009-11-10T23:00:10Z", 25.2]! ]! }! ]! }! ]! }! QUERY: RESULTS:
  32. SELECT value FROM cpu! WHERE host = ‘serverA'OR host =

    'serverB'! QUERY: {! "series": [! {! "name": "cpu",! "tags": {! "host": "serverA"! },! "columns": ["time", "value"],! "values": []! },! {! "name": "cpu",! "tags": {! "host": "serverB"! },! "columns": ["time", "value"],! "values": []! } ! ]! }! SERIES! IN RESULT:
  33. SELECT percentile(90, value) FROM cpu! WHERE time > now() -

    4h! GROUP BY time(10m), region QUERY: [! {! "name": "cpu",! "tags": {! "region": "us-west"! },! "columns": ["time", "percentile"],! "values": []! },! {! "name": "cpu",! "tags": {! "region": "us-east"! },! "columns": ["time", "percentile"],! "values": []! } ! ]! SERIES! IN RESULT:
  34. Multiple aggregates SELECT mean(value), percentile(90, value), min(value), max(value)! FROM cpu!

    WHERE host='serverA' AND time > now() - 48h! GROUP BY time(1h)!
  35. Return every series in CPU SELECT mean(value)! FROM cpu! WHERE

    time > now() - 48h! GROUP BY time(1h), *!
  36. Discovery based on tags

  37. {! "results":[! {! "query": "SHOW MEASUREMENTS",! "series": [! {! "name":

    "measurements",! "columns": ["name"],! "values": [! ["cpu"],! ["memory"],! ["network"]! ]! }! ]! }! ]! }!
  38. {! "results":[! {! "query": "SHOW SERIES",! "series": [! {! "name":

    "cpu",! "columns": ["id", "region", "host"],! "values": [! [1, "us-west", "serverA"],! [2, "us-east", "serverB"]! ]! }! ]! }! ]! }!
  39. {! "query": "SHOW MEASUREMENTS WHERE service='redis'",! "series": [! {! "name":

    "measurements",! "name": "series",! "columns": ["measurement"],! "values": [! ["key_count"],! ["connections"]! ]! }! ]! }!
  40. {! "query": "SHOW TAG KEYS from cpu",! "series": [! {!

    "name": "keys",! "columns": ["key"],! "values": [! ["region"],! ["host"]! ]! }! ]! }!
  41. {! "query": "SHOW TAG VALUES WITH KEY = service",! "series":

    [! {! "name": "series",! "columns": ["service"],! "values": [! ["redis"],! ["apache"]! ]! }! ]! }!
  42. {! "query": "SHOW TAG VALUES FROM cpu WITH KEY =

    service",! "series": [! {! "name": "series",! "columns": ["service"],! "values": [! ["redis"],! ["apache"]! ]! }! ]! }!
  43. Much more • Retention policies • Automatic downsampling and aggregation

    • Clustering
  44. onto the rewrite…

  45. Since November we’ve been rewriting InfluxDB from scratch.

  46. – Joel Spolsky on rewriting from scratch Things You Should

    Never Do, Part I http://www.joelonsoftware.com/articles/fog0000000069.html “… the single worst strategic mistake that any software company can make …”
  47. Why would we rewrite?

  48. –InfluxDB design philosophy “Optimize for developer happiness.”

  49. What does this mean?

  50. for InfluxDB users…

  51. simple setup

  52. flexible API

  53. empowering API

  54. performant by default

  55. helps developers build analytics applications faster

  56. for InfluxDB developers…

  57. fast build time

  58. idiomatic Go code

  59. easy to build and contribute

  60. Not Happy

  61. Split across three different legitimate areas

  62. Feature Requests Moving average, different kinds of derivatives, ways to

    fill data, top N for a given period, exact data point for min/max
  63. Performance Bugs High CPU load on query, where clause takes

    too long, list series takes too long
  64. Bugs open file handles, out of memory, clustering crashes

  65. Questions (illegitimate?)

  66. Some questions point to bad API design

  67. We identified 3 different areas of improvement

  68. • API Design • Structural Design Problems Limiting Performance •

    Underlying Technology Choices Causing Bugs
  69. API Design should be understandable, should push users in the

    right direction
  70. Previous API Many series with metadata in the name like

    in Graphite region.us.data_center.1.host.serverA.network_in! region.us.data_center.1.host.serverA.network_out! 5m.mean.region.us.data_center.1.host.serverA.network_in! 5m.mean.region.us.data_center.1.host.serverA.network_out!
  71. Understandable API Design: Retention Policies • Previously called shard spaces

    • Users tell the server which shard space to read/ write data into based on a regex to match against the series name
  72. Too hard to use, couldn’t tell where data was going

  73. Users could accidentally hide a bunch of data that was

    previously available
  74. Solution: have the user be explicit at write or query

    time
  75. Pushing users in the right direction: Tags SELECT mean(value) FROM

    cpu! WHERE host = 'serverA'! Users wanted this:
  76. Columns • Wasted space • Add Indexes? • User still

    has to create them
  77. Everyone was expecting tags… Indexed metadata and series

  78. Structural Design Problems LIST SERIES /.*region\.uswest.*/! Tell users to have

    many series: SELECT mean(value)! FROM merge(/.*region\.uswest.*/)! WHERE time > now() - 2h! GROUP BY time(5m)!
  79. No way to make that perform well with > 100k

    series
  80. Solution: measurements and tags break up the namespace

  81. Structural Design Problems: Query Engine • Pipes raw data over

    the network • Need to redesign to get data locality • MapReduce framework
  82. Solution: refactor/rewrite the query engine

  83. Underlying Technology Choices • Protobufs • Performance • Everywhere in

    code (network, database)
  84. Solution: switch to raw bytes for storage

  85. Underlying Technology Choices • LevelDB • Too many file handles

    • No online backups • Too hard to transfer shard from one server to another
  86. Solution: switch to BoltDB

  87. Underlying Technology Choices • Flex & Bison for Parser •

    Very hard to understand and update • CGo code
  88. Solution: switch to pure Go parser

  89. All of these things together pointed to a rewrite Large

    API changes, underlying technology changes, and code in every area getting touched
  90. A refactor would have been a rewrite

  91. It would have dragged breaking changes out over multiple releases

  92. With a rewrite we rip the bandaid off quickly

  93. –InfluxDB design philosophy “Optimize for developer happiness.”

  94. The 0.9.0 release gives us a solid foundation to build

    on
  95. Thank you Paul Dix @pauldix paul@influxdb.com ! P.S. we’re hiring

    Go and front-end developers