$30 off During Our Annual Pro Sale. View Details »

HeteroTSDB: An Extensible Time Series Database for Automatically Tiering on Heterogeneous Key-Value Stores

HeteroTSDB: An Extensible Time Series Database for Automatically Tiering on Heterogeneous Key-Value Stores

Yuuki Tsubouchi, Asato Wakisaka, Ken Hamada, Masayuki Matsuki, Hiroshi Abe, and Ryosuke Matsumoto, HeteroTSDB: An Extensible Time Series Database for Automatically Tiering on Heterogeneous Key-Value Stores, at 43rd IEEE International Conference Computer Software & Applications (COMPSAC), 2019.

paper: https://yuuk.io/papers/transtracer_dicomo2019.pdf

Monitoring service providers have arisen to meet increasing demands from system administrators. Providing a monitoring service requires high-resolution time series, long- term retention, and high write scalability to a time series database (TSDB). Nevertheless, no research to date has exam- ined proposed TSDBs with consideration of the extensibility of data structures for function additions of a monitoring service. As described herein, we introduce a TSDB architecture for automatically tiering on heterogeneous key-value stores (KVS) that combines with an in-memory KVS and an on-disk KVS. Our proposed architecture, by unbundling the data structure on memory and disk as different KVSs, enables the TSDB extensibility to duplicate writes to new data structures optimized for each use case without reducing writing efficiency or data storage efficiency.

Yuuki Tsubouchi (yuuk1)

July 16, 2019
Tweet

More Decks by Yuuki Tsubouchi (yuuk1)

Other Decks in Research

Transcript

  1. SAKURA internet Inc.
    (C) Copyright 1996-2019 SAKURA internet Inc
    SAKURA Internet
    Research Center
    HeteroTSDB: An Extensible
    Time Series Database for Automatically Tiering
    on Heterogeneous Key-Value Stores
    July 16 2019 Yuuki Tsubouchi
    Yuuki Tsubouchi, Asato Wakisaka, Ken Hamada, Masayuki Matsuki,
    Hiroshi Abe and Ryosuke Matsumoto
    43rd IEEE International Conference Computer Software & Applications (COMPSAC)

    View Slide

  2. 2
    1. Introduction
    2. Related Works
    3. Proposed Method
    4. Evaluation
    5. Conclusion
    Agenda

    View Slide

  3. 1.
    Introduction

    View Slide

  4. 4
    ɾDemand to internet services is increasing for availability and
    response speed.
    ɾA dedicated system for monitoring is necessary to identify a
    problem quickly in the systems.
    ɾHowever, it brings additional system setup and update work.
    ɾMany system administrators use a monitoring service to reduce
    these burdens.
    ɾA monitoring service builds a Time Series Database(TSDB) to store
    the time series data for monitoring.
    Background

    View Slide

  5. 5
    ɾA monitoring system periodically collects metrics for measuring the
    state of systems.
    ɾrecords the metrics in a TSDB with high resolution
    ɾretains past data for a long time
    ɾThe challenges for TSDB
    ɾWrite processing efficiency
    ɾData storage efficiency
    The existing challenges for TSDB
    TSDB
    Heavy
    writing
    Long-term
    storage

    View Slide

  6. 6
    ɾA monitoring service requires the addition of new functions.
    ɾto analyze long-term metrics for data warehouse
    ɾto label metrics by inverted index for flexible searching
    ɾIt is necessary for TSDB extensibility for data structure.
    ɾThe existing TSDBs configure a DBMS as a single building block, so
    that we should change the DBMS software itself to add a new data
    structure.
    ɾHowever, it is difficult to extend a DBMS with tightly coupled
    functions.
    The new challenge for TSDB

    View Slide

  7. 7
    HeteroTSDB architecture
    ɾThe extensibility of the data structure
    ↪ Loosely-coupled architecture with heterogeneous DBMSs
    1. separate the single TSDB architecture into some parts such as log,
    index, time series data structure
    2. add a standalone DBMS per an additional data structure
    3. combine the parts as one loosely-coupled TSDB
    DBMS
    Single
    TSDB
    DBMS
    Index
    DBMS
    Log
    HeteroTSDB
    Time series data

    View Slide

  8. 2.
    Related Works

    View Slide

  9. 9
    What is a time series database?
    ɾA DBMS that has the following properties is called TSDB.
    (i) store a row of data that consists of timestamp, value, and optional
    tags(e.g., name)
    (ii) store multiple rows of time series data grouped together
    (iii) can query for rows of data
    (iv) can contain a timestamp or a time range in a query
    Quoted from A. Bader, O. Kopp, and F. Michael, “Survey and Comparison of Open Source Time Series Databases,” in
    Datenbanksysteme für Business, Technologie und Web (BTW) - Workshopband. GI, 2017, pp. 249–268.

    View Slide

  10. 10
    The existing TSDBs
    ɾA TSDB with a general-purpose storage
    ɾIt can store data excepting time series data
    ɾe.g., RDBMS, NoSQL(Wide column store, Document store, Object
    storage)…
    ɾOpenTSDB[16]
    ɾA TSDB with a storage specialized for TSDB workload [17,10]
    ɾMaximizing processing and storage efficiency for TSDB workload
    ɾGorilla[17], InfluxDB[10]
    [16] OpenTSDB, http://opentsdb.net.
    [17] Pelkonen T et al., “Gorilla: A Fast, Scalable, In-Memory Time Series Database,” 41st International Conference on VLDB, vol.8, no.12, pp.1816–1827, 2015.
    [10] InfluxData, InfluxDB, https://www.influxdata.com/time-series-platform/influxdb/.

    View Slide

  11. 11
    1. A service provider should change the DBMS software itself to add a
    new data structure.
    ɾIt is difficult to extend the DBMS with tightly coupled functions.
    2. A service provider are forced to learn, build and update TSDBs with a
    storage engine specialized for TSDB workload.
    ɾOpenTSDB that depends on timestamp sorting on HBase.
    ɾThe system administrators pay that operational costs.
    The problems with the existing methods

    View Slide

  12. 3.
    Proposed Method

    View Slide

  13. 13
    HeteroTSDB architecture
    1. The extensibility of the data structure
    ↪ Loosely-coupled architecture with heterogeneous DBMSs
    2. Reducing the operational costs of the DBMS specialized for TSDB
    ↪ Compatibility by key-value-based time series data structure
    3. Improving the efficiency of write processing and data storage
    ↪ Automatically tiering between in-memory KVS and on-disk KVS
    In-Memory
    KVS
    On—Disk
    KVS
    Tiering
    Single
    TSDB
    DBMS
    Index
    DBMS
    Log
    Our TSDB architecture

    View Slide

  14. 14
    High-level architecture overview
    Message
    Broker
    Metric
    Writer
    In-Memory
    KVS
    Client
    On—Disk
    KVS
    Metric
    Mover
    (1) write (2) subscribe and write (3) migration
    Metric
    Reader
    (i) query
    (iii)
    merge data points
    (ii)
    read from each KVS
    (ii)
    1. How can we extend data structure?
    2. How do we model our time-series data structure as key-value format?
    3. How do we tier between an in-memory KVS and an on-disk KVS?
    Log

    View Slide

  15. 1. How to extend data structure?

    View Slide

  16. 16
    1. How to extend data structure?
    Message
    Broker
    Metric
    Writer
    In-Memory
    KVS
    Client
    On—Disk
    KVS
    DBMSs
    Metric
    Mover
    DBMSs
    Stream replication Batch replication
    ɾPros: The light writing workload to the
    additional DBMSs
    ɾCons: Lookup is delayed
    ɾPros: Immediate lookup
    ɾCons: The additional DBMSs are
    exposed to the heavy writing
    workload
    To guarantee consistency between the KVS and the additional DBMSs,
    their data structure should be idempotent.

    View Slide

  17. 2. How to model the data structure as
    key-value?

    View Slide

  18. 18
    2. How to model the data structure as key-value?
    name:wtimestamp
    Record1
    Key Value
    List of a pair of timestamp and value
    Record2
    1. Reading efficiently time series
    ɾstore multiple data points in the same series in one record.
    ɾdivide the series into fixed-width time windows.
    Time window
    name:wtimestamp List of a pair of timestamp and value

    View Slide

  19. 19
    2. How to model the data structure as key-value?
    2. Range searching time series
    name:wtimestamp
    Key Value
    name:wtimestamp
    ɾAligning the timestamp to a multiple of the resolution before
    writing data points
    ɾ1482703011 (unix time) → 1482703020 (Multiple of 60 secs)
    Start
    End
    wtimestamp1
    wtimestamp2
    wtimestampN
    ɾ
    ɾ
    ɾ
    ɾ
    ……
    ……

    View Slide

  20. 20
    2. How to model the data structure as key-value?
    name:wtimestamp
    Key Value
    Hash map of a pair of timestamp and value
    3. Preserving idempotent when rewriting the same datapoint
    ɾUse hash map
    { timestamp: value, …}
    name:wtimestamp

    View Slide

  21. 3. How to automatically tier
    between an in-memory KVS and an
    on-disk KVS?

    View Slide

  22. 22
    3. How to automatically tier between KVSs?
    1. The data points are written to the in-memory KVS.
    2. The old data points on the in-memory KVS is moved to the on-disk
    KVS in units of time series.
    ɾIf the in-memory KVS breaks down, it is recovered from write-ahead
    log(WAL).
    ɾA message broker implements WAL.
    ɾWe propose following two methods for data movement.
    ɾTimer method
    ɾCounting method
    In-Memory
    KVS
    On—Disk
    KVS
    Tiering
    Message
    broker
    Log

    View Slide

  23. 23
    Data movement: Timer Method
    KVS KVS
    Record Record
    Trigger
    Record Trigger Record
    Metric
    Writer
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    1. A timer is set in units of record on the KVS.
    2. When the timer becomes 0, the KVS kicks a trigger.
    3. The trigger inserts the record to the another KVS.
    1.
    1.
    2.
    2.
    3.
    3.
    Timer (TTL)

    View Slide

  24. 24
    Data movement: Counting Method
    Metric
    Writer
    In-Memory KVS
    1. Count data points when writing
    2. Read the batch of data points if the
    number is over a certain number
    3. Write the batch of data points
    On-Disk KVS
    At regular intervals
    Counting method is for KVS that doesn’t implement TTL

    View Slide

  25. 25
    Summary of HeteroTSDB architecture
    1. Loosely-coupled architecture with heterogeneous DBMSs
    ɾStream replication and Batch replication
    2. Compatibility by key-value-based time series data structure
    ɾTime window with fixed width
    ɾIdempotent data structure by Hash map
    3. Automatically tiering between in-memory KVS and on-disk KVS
    ɾTimer method and Counting method
    In-Memory
    KVS
    On—Disk
    KVS
    Tiering
    Single
    TSDB
    DBMS
    Index
    DBMS
    Log
    Our TSDB architecture

    View Slide

  26. 4.
    Evaluation

    View Slide

  27. 27
    Evaluation
    1. Implementation
    2. Performance Evaluation
    ɾThe number of writes per minute
    ɾCPU and Memory resource usage
    3. Experience in production
    ɾPlease see the more details in our paper.

    View Slide

  28. 1. Implementation

    View Slide

  29. 29
    ɾUse the serverless platform on AWS
    ɾ3-tier: Memory - SSD - HDD
    Our implementation

    View Slide

  30. 2. Performance Evaluation

    View Slide

  31. 31
    3PMF *UFN 4QFDJpDBJUPO
    #FODINBSL
    $MJFOU
    &$JOTUBODF
    *OTUBODF5ZQF
    04
    ,FSOFM
    DYMBSHF
    "NB[PO-JOVY

    .FTTBHF
    #SPLFS
    ,JOFTJT%BUB4USFBNT
    /VNCFSPGTIBSET
    .FUSJD
    8SJUFS
    -BNCEB

    .FNPSZ
    3VOUJNF
    .#

    /PEFKT
    *O.FNPSZ
    ,74
    &MBTUJ$BDIFGPS3FEJT
    /PEFUZQF
    /VNCFSPGTIBSET
    DBDIFSMBSHF

    0O%JTL,74 %ZOBNP%# 8$6
    The benchmarker’s parameters
    • Number of agents: 40,000
    • Interval of data points
    transmission: 1 min
    • 100 data points / transmission
    We have observed changes for
    2 hours.

    View Slide

  32. 32
    The number of writes per minute
    0
    1
    2
    3
    4
    5
    0 20 40 60 80 100 120
    datapoint writes / min (mega)
    minutes
    In-Memory KVS
    On-Disk KVS
    In-memory KVS: The number of
    writes per minute was constant at a
    value of about 4 M.
    On-disk KVS: The number of times of
    writing to the on-disk KVS took a value
    between 70k and 170k.
    1/20
    The number of writing per minute to
    the on-disk KVS is smaller to about
    1/20 than that to the in-memory KVS.

    View Slide

  33. 33
    Resource usage of in-memory KVS
    0
    10
    20
    30
    40
    50
    60
    70
    80
    90
    100
    0 20 40 60 80 100 120
    0
    2
    4
    6
    8
    10
    12
    14
    16
    CPU usage (%)
    Free memory size (GB)
    minutes
    master CPU usage (%)
    slave1 CPU usage (%)
    slave2 CPU usage (%)
    Free memory size (GB)
    Free memory size transits
    while taking a constant
    value at around 10.5 GB.
    CPU usage transits while
    taking about 60~75%
    between the waveform
    shapes.

    View Slide

  34. 34
    ɾThe number of writes per minute
    ɾWithout in-memory KVS, on-disk KVS should accept the same
    number of write times with the in-memory KVS.
    ɾHowever, HeteroTSDB reduces the number of writes to the on-
    disk KVS to about 1/20 of them directly to the on-disk KVS.
    ɾResource usage of in-memory KVS
    ɾThe memory usage on the in-memory KVS is expected to
    increase monotonically with time.
    ɾHowever, HeteroTSDB remains constant with the memory usage
    of the in-memory KVS.
    Consideration for performance

    View Slide

  35. 3. Experience in production
    Deploying our architecture to the production of the monitoring
    service from August 2017 through August 2018.

    View Slide

  36. 36
    Faults in production
    ɾ(1) The loss of data points on the in-memory KVS
    ɾThe data points that remained in the message broker were
    restored by running the MetricWriter.
    ɾ(2) The temporary write error by a software bug

    View Slide

  37. 37
    Faults in production (1)
    ɾ(1) The loss of data points on the in-memory KVS
    ɾThe write load concentrated on a specific node of the in-memory
    KVS cluster.
    ɾThe memory consumption in the node reached the upper limit.
    ɾAs a result of the OS forcibly stopping the process, data of a
    plurality of specific metrics temporarily disappeared.
    ɾThe data points that remained in the message broker were
    restored by running the MetricWriter.

    View Slide

  38. 38
    Faults in production (2)
    ɾ(2) The permanent write error by a software bug.
    ɾ Too many data points with the same metric name and the same
    timestamp were written in a short time.
    ɾThe write query size for the in-memory KVS exceeds the upper
    limit of implementation.
    ɾThe Lambda function returned errors, was continued to be retried,
    and processing of the entire MetricWriter was delayed.
    ɾWe modified the program of MetricWriter to eliminate duplication
    of data points before writing.

    View Slide

  39. 5.
    Conclusion

    View Slide

  40. 40
    Conclusion
    ɾWe have proposed the TSDB architecture focused on extensibility,
    compatibility, and efficiency.
    ɾWe have implemented HeteroTSDB on an AWS serverless platform
    to reduce the burden of construction of multiple DBMSs.
    ɾThe results of our experiment
    ɾHeteroTSDB reduces the number of writes to the on-disk KVS to
    about 1/20 and remains the constant memory
    ɾHeteroTSDB remains constant with the memory usage of the in-
    memory KVS.

    View Slide

  41. 41
    Future Work
    ɾDemonstrating that the HeteroTSDB is practical in terms of
    performance after comparison with other TSDBs.
    ɾImplementing extensions of the data structure and evaluate the
    extensibility.
    ɾEvaluating the compatibility and show the limitation of
    HeteroTSDB.

    View Slide