Slide 1

Slide 1 text

Distributed Timeseries Database in Go

Slide 2

Slide 2 text

Go is the future of NoSQL/NewSQL

Slide 3

Slide 3 text

Databases written in Go 4 Prometheus 4 CockroachDb 4 InfluxDb 4 Dgraph 4 EtcD 4 Consuld

Slide 4

Slide 4 text

Talk about architecture

Slide 5

Slide 5 text

What is a timeseries

Slide 6

Slide 6 text

Use cases for timeseries 4 Stocks 4 Monitoring 4 IOT

Slide 7

Slide 7 text

About timeseries 4 Timeseries can be lossy 4 Timeseries compress uniquely on data sets 4 Write heavy 4 Key, Time, DataPoint 4 CNX:IND, June 15 12:23, $23.40

Slide 8

Slide 8 text

Dark days 4 Graphite 4 InfluxDb 4 Mysql storing metrics 4 OpenTSDB (UGGHHHHHH)

Slide 9

Slide 9 text

Prometheus

Slide 10

Slide 10 text

No content

Slide 11

Slide 11 text

rate(http_request_latency[1m])

Slide 12

Slide 12 text

No content

Slide 13

Slide 13 text

Initial architecture Beta for 3000 customers

Slide 14

Slide 14 text

Hash sharded Prometheus 3-4 per datacenter

Slide 15

Slide 15 text

No content

Slide 16

Slide 16 text

No content

Slide 17

Slide 17 text

Performance requirements 4 3 Gbits/sec of traffic 4 100k Writes a second 4 50ms Reads 4 100,000 customers to start 4 20 TB of storage

Slide 18

Slide 18 text

Introducing Vulcan https://github.com/digitalocean/vulcan

Slide 19

Slide 19 text

Strange PRs

Slide 20

Slide 20 text

A fateful meeting at Soundcloud...

Slide 21

Slide 21 text

Architecture changes 4 Split to microservices 4 Containerization 4 Message Queues

Slide 22

Slide 22 text

Pipelining data

Slide 23

Slide 23 text

No content

Slide 24

Slide 24 text

Scaling storage

Slide 25

Slide 25 text

No content

Slide 26

Slide 26 text

Metrics format

Slide 27

Slide 27 text

Timeseries Schema 4 V1 Timeseries Table 4 key (Combined Key) 4 timestamp (Combined Key) 4 datapoint (float64)

Slide 28

Slide 28 text

4 V2 Chunks (1 KB)

Slide 29

Slide 29 text

4 V2 Timeseries Table 4 key (Combined Key) 4 timestamp range (2hours) (Combined Key) 4 raw data (1kb blob)

Slide 30

Slide 30 text

4 Index Table 4 Customer (Combined key) 4 keyPrefix (Combined key) 4 time 4 key

Slide 31

Slide 31 text

In memory query engine

Slide 32

Slide 32 text

Downsampling

Slide 33

Slide 33 text

Final Architecture

Slide 34

Slide 34 text

No content

Slide 35

Slide 35 text

Questions?