Refactoring InfluxDB: from Go to Go

Slide 1

Slide 1 text

Refactoring InfluxDB: from Go to Go Paul Dix CEO and cofounder of InfluxDB @pauldix paul@influxdb.com

Slide 2

Slide 2 text

About me…

Slide 3

Slide 3 text

No content

Slide 4

Slide 4 text

Organizer NYC Machine Learning (5,500+ members)

Slide 5

Slide 5 text

Series Editor - “Data & Analytics”

Slide 6

Slide 6 text

Cofounder & CEO of InﬂuxDB

Slide 7

Slide 7 text

Onto the talk!

Slide 8

Slide 8 text

Confession… (not a Go talk)

Slide 9

Slide 9 text

API Design

Slide 10

Slide 10 text

Software Design

Slide 11

Slide 11 text

–InﬂuxDB design philosophy “Optimize for developer happiness.”

Slide 12

Slide 12 text

Refactor Rewrite

Slide 13

Slide 13 text

Before that, what’s InﬂuxDB?

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Open source distributed time series database written in Go with no external dependencies

Slide 16

Slide 16 text

Metrics

Slide 17

Slide 17 text

Time Series

Slide 18

Slide 18 text

Analytics

Slide 19

Slide 19 text

Events

Slide 20

Slide 20 text

Use Cases • Metrics (DevOps) • Sensor Data • Realtime Analytics

Slide 21

Slide 21 text

Features Upcoming 0.9.0 release

Slide 22

Slide 22 text

Data model • Databases

Slide 23

Slide 23 text

Data model • Databases • Measurements • cpu_load, temperature, log_lines, click, etc.

Slide 24

Slide 24 text

Data model • Databases • Measurements • cpu_load, temperature, log_lines, click, etc. • Tags • region=uswest, host=serverA, building=23, service=redis, etc.

Slide 25

Slide 25 text

Slide 26

Slide 26 text

Data model • Databases • Measurements • cpu_load, temperature, log, click, etc. • Tags • region=uswest, host=serverA, building=23, service=redis, etc. • Series - measurement + unique tagset • Points • Fields - bool, int64, ﬂoat64, string, []byte • Timestamp - nano epoch

Slide 27

Slide 27 text

Writing Data curl -XPOST 'http://localhost:8086/write' -d '...'!

Slide 28

Slide 28 text

Writing Data {! "database": "mydb",! "retentionPolicy": "30d",! "points": [! {! "name": "cpu_load",! "tags": {! "host": "server01",! "region": "us-west"! },! "timestamp": "2009-11-10T23:00:00Z",! "fields": {! "value": 0.64! }! }! ]! }! Measurement Tags Fields

Slide 29

Slide 29 text

Querying curl -G 'http://localhost:8086/query' --data-urlencode "q=..."!

Slide 30

Slide 30 text

SQL-ish query language

Slide 31

Slide 31 text

SELECT value FROM cpu WHERE host = 'serverA'! {! "results":[! {! "query": "SELECT value FROM cpu WHERE host='serverA'",! "series": [! {! "name": "cpu",! "tags": {! "host": "serverA"! },! "columns": ["time", "value"],! "values": [! ["2009-11-10T23:00:00Z", 22.1],! ["2009-11-10T23:00:10Z", 25.2]! ]! }! ]! }! ]! }! QUERY: RESULTS:

Slide 32

Slide 32 text

SELECT value FROM cpu! WHERE host = ‘serverA'OR host = 'serverB'! QUERY: {! "series": [! {! "name": "cpu",! "tags": {! "host": "serverA"! },! "columns": ["time", "value"],! "values": []! },! {! "name": "cpu",! "tags": {! "host": "serverB"! },! "columns": ["time", "value"],! "values": []! } ! ]! }! SERIES! IN RESULT:

Slide 33

Slide 33 text

SELECT percentile(90, value) FROM cpu! WHERE time > now() - 4h! GROUP BY time(10m), region QUERY: [! {! "name": "cpu",! "tags": {! "region": "us-west"! },! "columns": ["time", "percentile"],! "values": []! },! {! "name": "cpu",! "tags": {! "region": "us-east"! },! "columns": ["time", "percentile"],! "values": []! } ! ]! SERIES! IN RESULT:

Slide 34

Slide 34 text

Multiple aggregates SELECT mean(value), percentile(90, value), min(value), max(value)! FROM cpu! WHERE host='serverA' AND time > now() - 48h! GROUP BY time(1h)!

Slide 35

Slide 35 text

Return every series in CPU SELECT mean(value)! FROM cpu! WHERE time > now() - 48h! GROUP BY time(1h), *!

Slide 36

Slide 36 text

Discovery based on tags

Slide 37

Slide 37 text

{! "results":[! {! "query": "SHOW MEASUREMENTS",! "series": [! {! "name": "measurements",! "columns": ["name"],! "values": [! ["cpu"],! ["memory"],! ["network"]! ]! }! ]! }! ]! }!

Slide 38

Slide 38 text

{! "results":[! {! "query": "SHOW SERIES",! "series": [! {! "name": "cpu",! "columns": ["id", "region", "host"],! "values": [! [1, "us-west", "serverA"],! [2, "us-east", "serverB"]! ]! }! ]! }! ]! }!

Slide 39

Slide 39 text

{! "query": "SHOW MEASUREMENTS WHERE service='redis'",! "series": [! {! "name": "measurements",! "name": "series",! "columns": ["measurement"],! "values": [! ["key_count"],! ["connections"]! ]! }! ]! }!

Slide 40

Slide 40 text

{! "query": "SHOW TAG KEYS from cpu",! "series": [! {! "name": "keys",! "columns": ["key"],! "values": [! ["region"],! ["host"]! ]! }! ]! }!

Slide 41

Slide 41 text

{! "query": "SHOW TAG VALUES WITH KEY = service",! "series": [! {! "name": "series",! "columns": ["service"],! "values": [! ["redis"],! ["apache"]! ]! }! ]! }!

Slide 42

Slide 42 text

{! "query": "SHOW TAG VALUES FROM cpu WITH KEY = service",! "series": [! {! "name": "series",! "columns": ["service"],! "values": [! ["redis"],! ["apache"]! ]! }! ]! }!

Slide 43

Slide 43 text

Much more • Retention policies • Automatic downsampling and aggregation • Clustering

Slide 44

Slide 44 text

onto the rewrite…

Slide 45

Slide 45 text

Since November we’ve been rewriting InﬂuxDB from scratch.

Slide 46

Slide 46 text

– Joel Spolsky on rewriting from scratch Things You Should Never Do, Part I http://www.joelonsoftware.com/articles/fog0000000069.html “… the single worst strategic mistake that any software company can make …”

Slide 47

Slide 47 text

Why would we rewrite?

Slide 48

Slide 48 text

–InﬂuxDB design philosophy “Optimize for developer happiness.”

Slide 49

Slide 49 text

What does this mean?

Slide 50

Slide 50 text

for InﬂuxDB users…

Slide 51

Slide 51 text

simple setup

Slide 52

Slide 52 text

ﬂexible API

Slide 53

Slide 53 text

empowering API

Slide 54

Slide 54 text

performant by default

Slide 55

Slide 55 text

helps developers build analytics applications faster

Slide 56

Slide 56 text

for InﬂuxDB developers…

Slide 57

Slide 57 text

fast build time

Slide 58

Slide 58 text

idiomatic Go code

Slide 59

Slide 59 text

easy to build and contribute

Slide 60

Slide 60 text

Not Happy

Slide 61

Slide 61 text

Split across three different legitimate areas

Slide 62

Slide 62 text

Feature Requests Moving average, different kinds of derivatives, ways to ﬁll data, top N for a given period, exact data point for min/max

Slide 63

Slide 63 text

Performance Bugs High CPU load on query, where clause takes too long, list series takes too long

Slide 64

Slide 64 text

Bugs open ﬁle handles, out of memory, clustering crashes

Slide 65

Slide 65 text

Questions (illegitimate?)

Slide 66

Slide 66 text

Some questions point to bad API design

Slide 67

Slide 67 text

We identiﬁed 3 different areas of improvement

Slide 68

Slide 68 text

• API Design • Structural Design Problems Limiting Performance • Underlying Technology Choices Causing Bugs

Slide 69

Slide 69 text

API Design should be understandable, should push users in the right direction

Slide 70

Slide 70 text

Previous API Many series with metadata in the name like in Graphite region.us.data_center.1.host.serverA.network_in! region.us.data_center.1.host.serverA.network_out! 5m.mean.region.us.data_center.1.host.serverA.network_in! 5m.mean.region.us.data_center.1.host.serverA.network_out!

Slide 71

Slide 71 text

Understandable API Design: Retention Policies • Previously called shard spaces • Users tell the server which shard space to read/ write data into based on a regex to match against the series name

Slide 72

Slide 72 text

Too hard to use, couldn’t tell where data was going

Slide 73

Slide 73 text

Users could accidentally hide a bunch of data that was previously available

Slide 74

Slide 74 text

Solution: have the user be explicit at write or query time

Slide 75

Slide 75 text

Pushing users in the right direction: Tags SELECT mean(value) FROM cpu! WHERE host = 'serverA'! Users wanted this:

Slide 76

Slide 76 text

Columns • Wasted space • Add Indexes? • User still has to create them

Slide 77

Slide 77 text

Everyone was expecting tags… Indexed metadata and series

Slide 78

Slide 78 text

Structural Design Problems LIST SERIES /.*region\.uswest.*/! Tell users to have many series: SELECT mean(value)! FROM merge(/.*region\.uswest.*/)! WHERE time > now() - 2h! GROUP BY time(5m)!

Slide 79

Slide 79 text

No way to make that perform well with > 100k series

Slide 80

Slide 80 text

Solution: measurements and tags break up the namespace

Slide 81

Slide 81 text

Structural Design Problems: Query Engine • Pipes raw data over the network • Need to redesign to get data locality • MapReduce framework

Slide 82

Slide 82 text

Solution: refactor/rewrite the query engine

Slide 83

Slide 83 text

Underlying Technology Choices • Protobufs • Performance • Everywhere in code (network, database)

Slide 84

Slide 84 text

Solution: switch to raw bytes for storage

Slide 85

Slide 85 text

Underlying Technology Choices • LevelDB • Too many ﬁle handles • No online backups • Too hard to transfer shard from one server to another

Slide 86

Slide 86 text

Solution: switch to BoltDB

Slide 87

Slide 87 text

Underlying Technology Choices • Flex & Bison for Parser • Very hard to understand and update • CGo code

Slide 88

Slide 88 text

Solution: switch to pure Go parser

Slide 89

Slide 89 text

All of these things together pointed to a rewrite Large API changes, underlying technology changes, and code in every area getting touched

Slide 90

Slide 90 text

A refactor would have been a rewrite

Slide 91

Slide 91 text

It would have dragged breaking changes out over multiple releases

Slide 92

Slide 92 text

With a rewrite we rip the bandaid off quickly

Slide 93

Slide 93 text

–InﬂuxDB design philosophy “Optimize for developer happiness.”

Slide 94

Slide 94 text

The 0.9.0 release gives us a solid foundation to build on

Slide 95

Slide 95 text

Thank you Paul Dix @pauldix paul@inﬂuxdb.com ! P.S. we’re hiring Go and front-end developers