Save 37% off PRO during our Black Friday Sale! »

IPC 2013 / WTC 2013: openTSDB

IPC 2013 / WTC 2013: openTSDB

This talk covers openTSDB, culture around monitoring. It looks into how openTSDB stores its data in HBase, which is based on HDFS.
I share our experiences and give advice on how to use this measuring tool

F6267239b86c21f830d51d87ff5bf5e5?s=128

Oliver Hankeln

October 30, 2013
Tweet

Transcript

  1. openTSDB - Metrics for a distributed world Oliver Hankeln /

    gutefrage.net @mydalon Mittwoch, 30. Oktober 13
  2. Who am I? Senior Engineer - Data and Infrastructure at

    gutefrage.net GmbH Was doing software development before DevOps advocate Mittwoch, 30. Oktober 13
  3. Who is Gutefrage.net? Germany‘s biggest Q&A platform #1 German site

    (mobile) about 5M Unique Users #3 German site (desktop) about 17M Unique Users > 4 Mio PI/day Part of the Holtzbrinck group Running several platforms (Gutefrage.net, Helpster.de, Cosmiq, Comprano, ...) Mittwoch, 30. Oktober 13
  4. What you will get Why we chose openTSDB What is

    openTSDB? How does openTSDB store the data? Our experiences Some advice Mittwoch, 30. Oktober 13
  5. Why we chose openTSDB Mittwoch, 30. Oktober 13

  6. We were looking at some options Munin Graphite openTSDB Ganglia

    Scales well no sort of yes yes Keeps all data no no yes no Creating metrics easy easy easy easy Mittwoch, 30. Oktober 13
  7. We have a winner! Munin Graphite openTSDB Ganglia Scales well

    no sort of yes yes Keeps all data no no yes no Creating metrics easy easy easy easy Bingo! Mittwoch, 30. Oktober 13
  8. Separation of concerns Mittwoch, 30. Oktober 13

  9. Separation of concerns UI was not important for our decision

    Alerting is not what we are looking for in our time series data base $ unzip|strip|touch|finger|grep|mount|fsck|more|yes| fsck|fsck|fsck|umount|sleep Mittwoch, 30. Oktober 13
  10. The ecosystem App feeds metrics in via RabbitMQ We base

    Icinga checks on the metrics We evaluate Skyline and Oculus by Etsy for anomaly detection We deploy sensors via chef Mittwoch, 30. Oktober 13
  11. openTSDB Written by Benoît Sigoure at StumbleUpon OpenSource (get it

    from github) Uses HBase (which is based on HDFS) as a storage Distributed system (multiple TSDs) Mittwoch, 30. Oktober 13
  12. The big picture HBase TSD TSD TSD TSD UI API

    tcollector This is really a cluster Mittwoch, 30. Oktober 13
  13. Putting data into openTSDB $ telnet tsd01.acme.com 4242 put proc.load.avg5min

    1382536472 23.2 host=db01.acme.com Mittwoch, 30. Oktober 13
  14. It gets even better tcollector is a python script that

    runs your collectors handles network connection, starts your collectors at set intervals does basic process management adds host tag, does deduplication Mittwoch, 30. Oktober 13
  15. A simple tcollector script #!/usr/bin/php <?php #Cast a die $die

    = rand(1,6); echo "roll.a.d6 " . time() . " " . $die . "\n"; Mittwoch, 30. Oktober 13
  16. What was that HDFS again? HDFS is a distributed filesystem

    suitable for Petabytes of data on thousands of machines. Runs on commodity hardware Takes care of redundancy Used by e.g. Facebook, Spotify, eBay,... Mittwoch, 30. Oktober 13
  17. Okay... and HBase? HBase is a NoSQL database / data

    store on top of HDFS Modeled after Google‘s BigTable Built for big tables (billions of rows, millions of columns) Automatic sharding by row key Mittwoch, 30. Oktober 13
  18. How openTSDB stores the data Mittwoch, 30. Oktober 13

  19. Keys are key! Data is sharded across regions based on

    their row key You query data based on the row key You can query row key ranges (say e.g. A...D) So: think about key design Mittwoch, 30. Oktober 13
  20. Take 1 Row key format: timestamp, metric id Mittwoch, 30.

    Oktober 13
  21. Take 1 Row key format: timestamp, metric id 1382536472, 5

    17 Server A Server B Mittwoch, 30. Oktober 13
  22. Take 1 Row key format: timestamp, metric id 1382536472, 5

    17 1382536472, 6 24 Server A Server B Mittwoch, 30. Oktober 13
  23. Take 1 Row key format: timestamp, metric id 1382536472, 5

    17 1382536472, 6 24 1382536472, 8 12 1382536473, 5 134 1382536473, 6 10 1382536473, 8 99 Server A Server B Mittwoch, 30. Oktober 13
  24. Take 1 Row key format: timestamp, metric id 1382536472, 5

    17 1382536472, 6 24 1382536472, 8 12 1382536473, 5 134 1382536473, 6 10 1382536473, 8 99 1382536474, 5 12 1382536474, 6 42 Server A Server B Mittwoch, 30. Oktober 13
  25. Solution: Swap timestamp and metric id Row key format: metric

    id, timestamp 5, 1382536472 17 6, 1382536472 24 8, 1382536472 12 5, 1382536473 134 6, 1382536473 10 8, 1382536473 99 5, 1382536474 12 6, 1382536474 42 Server A Server B Mittwoch, 30. Oktober 13
  26. Solution: Swap timestamp and metric id Row key format: metric

    id, timestamp 5, 1382536472 17 6, 1382536472 24 8, 1382536472 12 5, 1382536473 134 6, 1382536473 10 8, 1382536473 99 5, 1382536474 12 6, 1382536474 42 Server A Server B Mittwoch, 30. Oktober 13
  27. Take 2 Metric ID first, then timestamp Searching through many

    rows is slower than searching through viewer rows. (Obviously) So: Put multiple data points into one row Mittwoch, 30. Oktober 13
  28. Take 2 continued 5, 1382608800 +23 +35 +94 +142 5,

    1382608800 17 1 23 42 5, 1382612400 +13 +25 +88 +89 5, 1382612400 3 44 12 2 Mittwoch, 30. Oktober 13
  29. Take 2 continued 5, 1382608800 +23 +35 +94 +142 5,

    1382608800 17 1 23 42 5, 1382612400 +13 +25 +88 +89 5, 1382612400 3 44 12 2 Row key Mittwoch, 30. Oktober 13
  30. Take 2 continued 5, 1382608800 +23 +35 +94 +142 5,

    1382608800 17 1 23 42 5, 1382612400 +13 +25 +88 +89 5, 1382612400 3 44 12 2 Row key Cell Name Mittwoch, 30. Oktober 13
  31. Take 2 continued 5, 1382608800 +23 +35 +94 +142 5,

    1382608800 17 1 23 42 5, 1382612400 +13 +25 +88 +89 5, 1382612400 3 44 12 2 Row key Cell Name Data point Mittwoch, 30. Oktober 13
  32. Where are the tags stored? They are put at the

    end of the row key Both tag names and tag values are represented by IDs Mittwoch, 30. Oktober 13
  33. The Row Key 3 Bytes - metric ID 4 Bytes

    - timestamp (rounded down to the hour) 3 Bytes tag ID 3 Bytes tag value ID Total: 7 Bytes + 6 Bytes * Number of tags Mittwoch, 30. Oktober 13
  34. Let‘s look at some graphs Mittwoch, 30. Oktober 13

  35. Busting some Myths Mittwoch, 30. Oktober 13

  36. Myth: Keeping Data is expensive Gartner found the price for

    enterprise SSDs at 1$/GB in 2013 A data point gets compressed to 2-3 Bytes A metric that you measure every second then uses disk space for 18.9ct per year. Usually it is even cheaper Mittwoch, 30. Oktober 13
  37. If your work costs 50$ per hour and it takes

    you only one minute to think about and configure your RRD compaction setting, you could have collected that metric on a second-by-second basis for 4.4 YEARS instead. Mittwoch, 30. Oktober 13
  38. Myth: the amount of metrics is too limited Don‘t confuse

    Graphite metric count with openTSBD metric count. 3 Bytes of metric ID = 16.7M possibilities 3 Bytes tag value ID = 16.7M possibilities => at least 280 T metrics (graphite counting) Mittwoch, 30. Oktober 13
  39. Cultural issues Mittwoch, 30. Oktober 13

  40. Tools shape culture shapes tools It is time for a

    new monitoring culture! Embrace machine learning! Monitor everything in your organisation! Throw of the shackles of fixed intervals! Come, join the revolution! Mittwoch, 30. Oktober 13
  41. Our experiences Mittwoch, 30. Oktober 13

  42. What works well We store about 200M data points in

    several thousand time series with no issues tcollector is decoupling measurement from storage Creating new metrics is really easy You are free to choose your rhythm Mittwoch, 30. Oktober 13
  43. Challenges The UI is seriously lacking no annotation support out

    of the box no meta data for time series Only 1s time resolution (and only 1 value/s/ time series) Mittwoch, 30. Oktober 13
  44. salvation is coming OpenTSDB 2 is around the corner millisecond

    precision annotations and meta data improved API improved UI Mittwoch, 30. Oktober 13
  45. Friendly advice Pick a naming scheme and stick to it

    Use tags wisely (not more than 6 or 7 tags per data point) Use tcollector wait for openTSDB 2 ;-) Mittwoch, 30. Oktober 13
  46. Questions? Please contact me: oliver.hankeln@gutefrage.net @mydalon I‘ll upload the slides

    and tweet about it Mittwoch, 30. Oktober 13