Better graphite storage with cyanite

Slide 1

Slide 1 text

BETTER GRAPHITE STORAGE WITH CYANITE PIERRE-YVES RITSCHARD @PYR #CASSANDRASUMMIT 0

Slide 2

Slide 2 text

@PYR CTO at exoscale, the safe home for your cloud applications Open source developer: pithos, cyanite, riemann, collectd… Recovering Operations Engineer

Slide 3

Slide 3 text

AIM OF THIS TALK Presenting graphite and its ecosystem Presenting cyanite Show-casing simplicity through cassandra

Slide 4

Slide 4 text

OUTLINE Graphite overview The problem with graphite Cyanite solutions & internals Looking forward

Slide 5

Slide 5 text

GRAPHITE OVERVIEW

Slide 6

Slide 6 text

FROM THE SITE Graphite does two things: 1. Store numeric time-series data 2. Render graphs of this data on demand http://graphite.readthedocs.org

Slide 7

Slide 7 text

SCOPE A metrics tool Not a complete monitoring solution Interacts with metric submission tools

Slide 8

Slide 8 text

WHY ARE METRICS IMPORTANT Outside the scope of this talk Narrowing the gap between map and territory

Slide 9

Slide 9 text

GRAPHITE COMPONENTS whisper carbon graphite-web

Slide 10

Slide 10 text

WHISPER RRD like storage library Written in python Each file contains different roll-up periods and an aggregation method

Slide 11

Slide 11 text

CARBON Asynchronous (twisted) TCP and UDP service to input time- series data Simple storage rules Split across several daemons

Slide 12

Slide 12 text

CARBON-CACHE Main carbon daemon Temporarily caches values to RAM Writes out to whisper

Slide 13

Slide 13 text

CARBON-AGGREGATOR Aggregates data and forwards to carbon-cache Less I/O strain on the filesystem At the expense of resolution

Slide 14

Slide 14 text

CARBON-RELAY Provides sharding and replication Forwards to appropriate carbon-cache processes based on a provided hashing method

Slide 15

Slide 15 text

GRAPHITE-WEB Simple Django-Based HTTP api Persists configuration to SQL Data query and manipulation through a very simple DSL Graph rendering Composer client interface to build graphs # # s u m C P U v a l u e s s u m S e r i e s ( " c o l l e c t d . w e b 0 1 . c p u - * " ) # # p r o v i d e m e m o r y p e r c e n t a g e a l i a s ( a s P e r c e n t ( w e b 0 1 . m e m . u s e d , s u m S e r i e s ( w e b 0 1 . m e m . * ) ) , " m e m p e r c e n t " )

Slide 16

Slide 16 text

SCREENSHOTS

Slide 17

Slide 17 text

SCREENSHOTS

Slide 18

Slide 18 text

ARCHITECTURE OVERVIEW

Slide 19

Slide 19 text

MODULARITY IN GRAPHITE Recently improved A module can implement a storage strategy for graphite-web Carbon modularity is a bit harder

Slide 20

Slide 20 text

THE GRAPHITE ECOSYSTEM A wealth of tools are now graphite compatible

Slide 21

Slide 21 text

STATSD Very popular metric service to integrate within applications. Aggregates events in n second windows Ships off to graphite s t a t s d . i n c r e m e n t ' s e s s i o n . o p e n ' s t a t s d . g a u g e ' s e s s i o n . a c t i v e ' , 3 7 0 s t a t s d . t i m i n g ' p d f . c o n v e r t ' , 3 2 0

Slide 22

Slide 22 text

COLLECTD Very popular collection daemon with a graphite destination Every conceivable system metrics A wealth of additional metric sources (such as a fast statsd server) < p l u g i n w r i t e _ g r a p h i t e > < c a r b o n > H o s t " g r a p h i t e - h o s t " < / c a r b o n > < / p l u g i n >

Slide 23

Slide 23 text

GRAPHITE-API Alternative to graphite-web Shares data manipulation code No persistence of configuration

Slide 24

Slide 24 text

GRAFANA Increasingly popular alternative to graphite-web, with graphite-api Inspired by the kibana project for logstash Optional persistence to elasticsearch for configuration

Slide 25

Slide 25 text

RIEMANN Distributed system monitoring solution ( d e f g r a p h ! ( g r a p h i t e { : h o s t " g r a p h i t e - s e r v e r " } ) ) ( s t r e a m s ( w h e r e ( s e r v i c e " h t t p . 4 0 4 " ) ( r a t e 5 g r a p h ! ) ) )

Slide 26

Slide 26 text

AND A LOT MORE syslog-ng logstash descartes tasseo jmxtrans

Slide 27

Slide 27 text

HIGH VALUE PROJECT Active and friendly developer community Growing ecosystem Very few contenders

Slide 28

Slide 28 text

THE PROBLEM WITH GRAPHITE

Slide 29

Slide 29 text

ESSENTIALY A SINGLE-HOST SOLUTION Built in a day where cacti reigned Innovative project at the time which decoupled collection from storage and display

Slide 30

Slide 30 text

THE WHISPER FILE FORMAT One file per data point Optimized for space, not speed Plenty of seeks Only shared storage option is NFS… In many ways can be seen as RRD in python

Slide 31

Slide 31 text

SCALING STRATEGIES Tacked on after the fact The decoupled architecture means that both graphite-web and carbon need upfront knowledge on the locations of shard

Slide 32

Slide 32 text

SCALING OVERVIEW

Slide 33

Slide 33 text

IT GETS A BIT HAIRY Cluster topology must be stored on all nodes Manual replication mechanism (through carbon-relay) Changing cluster topology means re-assigning shards by hand

Slide 34

Slide 34 text

WHAT GRAPHITE CAN KEEP Persistence of configuration Local data manipulation

Slide 35

Slide 35 text

WHAT GRAPHITE WOULD NEED Automatic shard assignment Replication Easy management Easy cluster topology changes (horizontal scalability)

Slide 36

Slide 36 text

THE CYANITE APPROACH Leveraging Apache Cassandra to store time-series Leveraging Graphite for the interface

Slide 37

Slide 37 text

A CASSANDRA-BACKED CARBON REPLACEMENT Written in clojure Async I/O No more whisper files Fast storage Horizontally scalable Interfaced with graphite-web through graphite-cyanite

Slide 38

Slide 38 text

CYANITE DUTIES Providing graphite-compatible input methods (carbon listeners) Providing a way to retrieve metric names and metric time- series Implemented as two protocols A metric-store A path-store The rest is up to the graphite eco-system, through graphite- cyanite The recommended companion is graphite-api

Slide 39

Slide 39 text

GETTING UP AND RUNNING A simple configuration file c a r b o n : h o s t : " 1 2 7 . 0 . 0 . 1 " p o r t : 2 0 0 3 r e a d t i m e o u t : 3 0 r o l l u p s : - p e r i o d : 6 0 4 8 0 r o l l u p : 1 0 - p e r i o d : 1 0 5 1 2 0 r o l l u p : 6 0 0 h t t p : h o s t : " 0 . 0 . 0 . 0 " p o r t : 8 0 8 0 l o g g i n g : l e v e l : i n f o f i l e s : - " / v a r / l o g / c y a n i t e / c y a n i t e . l o g " s t o r e : c l u s t e r : ' l o c a l h o s t ' k e y s p a c e : ' m e t r i c '

Slide 40

Slide 40 text

GRAPHITE-CYANITE with graphite-web: S T O R A G E _ F I N D E R S = ( ' c y a n i t e . C y a n i t e F i n d e r ' , ) C Y A N I T E _ U R L S = ( ' h t t p : / / h o s t : p o r t ' , ) with graphite-api: c y a n i t e : u r l s : - h t t p : / / c y a n i t e - h o s t : p o r t f i n d e r s : - c y a n i t e . C y a n i t e F i n d e r

Slide 41

Slide 41 text

LEADING ARCHITECTURE DRIVERS Simplicity Optimize for speed As few moving parts as possible Multi-tenancy Resource efficiency Remain compatible with the graphite ecosystem

Slide 42

Slide 42 text

CYANITE INTERNALS

Slide 43

Slide 43 text

CASSANDRA IS GREAT FOR TIME-SERIES It bears repeating High write to read ratio workload No manual shard allocation or reassignment Sorted wide columns mean efficient retrieval of data

Slide 44

Slide 44 text

A NEW STACK

Slide 45

Slide 45 text

SIMPLE SCHEMA C R E A T E T A B L E " m e t r i c " ( t e n a n t t e x t , p e r i o d i n t , r o l l u p i n t , p a t h t e x t , t i m e b i g i n t , d a t a l i s t < d o u b l e > , P R I M A R Y K E Y ( ( t e n a n t , p e r i o d , r o l l u p , p a t h ) , t i m e ) )

Slide 46

Slide 46 text

TAKING ADVANTAGE OF WIDE COLUMNS

Slide 47

Slide 47 text

LOOKING FORWARD

Slide 48

Slide 48 text

REPLACING MORE GRAPHITE PARTS, EXTENDING FUNCTIONALITY Implement graphite's data manipulation functions Remove the need for graphite-api or graphite-web when using grafana Finish providing multi-tenancy options

Slide 49

Slide 49 text

PICKLE SUPPORT Easier integration in existing architectures Would allow integration with carbon-relay

Slide 50

Slide 50 text

ALTERNATIVE INPUT METHODS Support queue input of metrics Collectd already supports shipping graphite data to Apache Kafka Support the statsd protocol directly

Slide 51

Slide 51 text

PROVIDE A CYANITE LIBRARY Easy, standard-compliant storage from JVM based applications

Slide 52

Slide 52 text

BATCH OPERATIONS Compactions of rolled up series Dynamic thresholds Great opportunity to leverage the cassandra & spark interaction

Slide 53

Slide 53 text

A FEW TAKE-AWAYS Cassandra enabled a quick-win in about 1100 lines of clojure Greatly simplified scaling strategy Building block for a lot more Good way to reduce technology creep if you're already using cassandra

Slide 54

Slide 54 text

THANKS ! Cyanite owes a lot to: Max Penet (@mpenet) for the great alia library Bruno Renie (@brutasse) for graphite-api, graphite-cyanite and the initial nudge Datastax for the awesome cassandra java-driver Its contributors Apache Cassandra obviously @pyr – #CassandraSummit