Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Better graphite storage with cyanite

Better graphite storage with cyanite

From cassandra summit 2014. Overview of the graphite ecosystem and the cyanite alternative storage engine for graphite metrics.

Pierre-Yves Ritschard

September 11, 2014

More Decks by Pierre-Yves Ritschard

Other Decks in Technology


  1. @PYR CTO at exoscale, the safe home for your cloud

    applications Open source developer: pithos, cyanite, riemann, collectd… Recovering Operations Engineer
  2. AIM OF THIS TALK Presenting graphite and its ecosystem Presenting

    cyanite Show-casing simplicity through cassandra
  3. FROM THE SITE Graphite does two things: 1. Store numeric

    time-series data 2. Render graphs of this data on demand http://graphite.readthedocs.org
  4. WHY ARE METRICS IMPORTANT Outside the scope of this talk

    Narrowing the gap between map and territory
  5. WHISPER RRD like storage library Written in python Each file

    contains different roll-up periods and an aggregation method
  6. CARBON Asynchronous (twisted) TCP and UDP service to input time-

    series data Simple storage rules Split across several daemons
  7. GRAPHITE-WEB Simple Django-Based HTTP api Persists configuration to SQL Data

    query and manipulation through a very simple DSL Graph rendering Composer client interface to build graphs # # s u m C P U v a l u e s s u m S e r i e s ( " c o l l e c t d . w e b 0 1 . c p u - * " ) # # p r o v i d e m e m o r y p e r c e n t a g e a l i a s ( a s P e r c e n t ( w e b 0 1 . m e m . u s e d , s u m S e r i e s ( w e b 0 1 . m e m . * ) ) , " m e m p e r c e n t " )
  8. MODULARITY IN GRAPHITE Recently improved A module can implement a

    storage strategy for graphite-web Carbon modularity is a bit harder
  9. STATSD Very popular metric service to integrate within applications. Aggregates

    events in n second windows Ships off to graphite s t a t s d . i n c r e m e n t ' s e s s i o n . o p e n ' s t a t s d . g a u g e ' s e s s i o n . a c t i v e ' , 3 7 0 s t a t s d . t i m i n g ' p d f . c o n v e r t ' , 3 2 0
  10. COLLECTD Very popular collection daemon with a graphite destination Every

    conceivable system metrics A wealth of additional metric sources (such as a fast statsd server) < p l u g i n w r i t e _ g r a p h i t e > < c a r b o n > H o s t " g r a p h i t e - h o s t " < / c a r b o n > < / p l u g i n >
  11. GRAFANA Increasingly popular alternative to graphite-web, with graphite-api Inspired by

    the kibana project for logstash Optional persistence to elasticsearch for configuration
  12. RIEMANN Distributed system monitoring solution ( d e f g

    r a p h ! ( g r a p h i t e { : h o s t " g r a p h i t e - s e r v e r " } ) ) ( s t r e a m s ( w h e r e ( s e r v i c e " h t t p . 4 0 4 " ) ( r a t e 5 g r a p h ! ) ) )
  13. ESSENTIALY A SINGLE-HOST SOLUTION Built in a day where cacti

    reigned Innovative project at the time which decoupled collection from storage and display
  14. THE WHISPER FILE FORMAT One file per data point Optimized

    for space, not speed Plenty of seeks Only shared storage option is NFS… In many ways can be seen as RRD in python
  15. SCALING STRATEGIES Tacked on after the fact The decoupled architecture

    means that both graphite-web and carbon need upfront knowledge on the locations of shard
  16. IT GETS A BIT HAIRY Cluster topology must be stored

    on all nodes Manual replication mechanism (through carbon-relay) Changing cluster topology means re-assigning shards by hand
  17. WHAT GRAPHITE WOULD NEED Automatic shard assignment Replication Easy management

    Easy cluster topology changes (horizontal scalability)

    more whisper files Fast storage Horizontally scalable Interfaced with graphite-web through graphite-cyanite
  19. CYANITE DUTIES Providing graphite-compatible input methods (carbon listeners) Providing a

    way to retrieve metric names and metric time- series Implemented as two protocols A metric-store A path-store The rest is up to the graphite eco-system, through graphite- cyanite The recommended companion is graphite-api
  20. GETTING UP AND RUNNING A simple configuration file c a

    r b o n : h o s t : " 1 2 7 . 0 . 0 . 1 " p o r t : 2 0 0 3 r e a d t i m e o u t : 3 0 r o l l u p s : - p e r i o d : 6 0 4 8 0 r o l l u p : 1 0 - p e r i o d : 1 0 5 1 2 0 r o l l u p : 6 0 0 h t t p : h o s t : " 0 . 0 . 0 . 0 " p o r t : 8 0 8 0 l o g g i n g : l e v e l : i n f o f i l e s : - " / v a r / l o g / c y a n i t e / c y a n i t e . l o g " s t o r e : c l u s t e r : ' l o c a l h o s t ' k e y s p a c e : ' m e t r i c '
  21. GRAPHITE-CYANITE with graphite-web: S T O R A G E

    _ F I N D E R S = ( ' c y a n i t e . C y a n i t e F i n d e r ' , ) C Y A N I T E _ U R L S = ( ' h t t p : / / h o s t : p o r t ' , ) with graphite-api: c y a n i t e : u r l s : - h t t p : / / c y a n i t e - h o s t : p o r t f i n d e r s : - c y a n i t e . C y a n i t e F i n d e r
  22. LEADING ARCHITECTURE DRIVERS Simplicity Optimize for speed As few moving

    parts as possible Multi-tenancy Resource efficiency Remain compatible with the graphite ecosystem
  23. CASSANDRA IS GREAT FOR TIME-SERIES It bears repeating High write

    to read ratio workload No manual shard allocation or reassignment Sorted wide columns mean efficient retrieval of data

    B L E " m e t r i c " ( t e n a n t t e x t , p e r i o d i n t , r o l l u p i n t , p a t h t e x t , t i m e b i g i n t , d a t a l i s t < d o u b l e > , P R I M A R Y K E Y ( ( t e n a n t , p e r i o d , r o l l u p , p a t h ) , t i m e ) )
  25. REPLACING MORE GRAPHITE PARTS, EXTENDING FUNCTIONALITY Implement graphite's data manipulation

    functions Remove the need for graphite-api or graphite-web when using grafana Finish providing multi-tenancy options
  26. ALTERNATIVE INPUT METHODS Support queue input of metrics Collectd already

    supports shipping graphite data to Apache Kafka Support the statsd protocol directly
  27. BATCH OPERATIONS Compactions of rolled up series Dynamic thresholds Great

    opportunity to leverage the cassandra & spark interaction
  28. A FEW TAKE-AWAYS Cassandra enabled a quick-win in about 1100

    lines of clojure Greatly simplified scaling strategy Building block for a lot more Good way to reduce technology creep if you're already using cassandra
  29. THANKS ! Cyanite owes a lot to: Max Penet (@mpenet)

    for the great alia library Bruno Renie (@brutasse) for graphite-api, graphite-cyanite and the initial nudge Datastax for the awesome cassandra java-driver Its contributors Apache Cassandra obviously @pyr – #CassandraSummit