HeteroTSDB: An Extensible Time Series Database for Automatically Tiering on Heterogeneous Key-Value Stores

Yuuki Tsubouchi, Asato Wakisaka, Ken Hamada, Masayuki Matsuki, Hiroshi Abe, and Ryosuke Matsumoto, HeteroTSDB: An Extensible Time Series Database for Automatically Tiering on Heterogeneous Key-Value Stores, at 43rd IEEE International Conference Computer Software & Applications (COMPSAC), 2019.

paper: https://yuuk.io/papers/transtracer_dicomo2019.pdf

Monitoring service providers have arisen to meet increasing demands from system administrators. Providing a monitoring service requires high-resolution time series, long- term retention, and high write scalability to a time series database (TSDB). Nevertheless, no research to date has exam- ined proposed TSDBs with consideration of the extensibility of data structures for function additions of a monitoring service. As described herein, we introduce a TSDB architecture for automatically tiering on heterogeneous key-value stores (KVS) that combines with an in-memory KVS and an on-disk KVS. Our proposed architecture, by unbundling the data structure on memory and disk as different KVSs, enables the TSDB extensibility to duplicate writes to new data structures optimized for each use case without reducing writing efficiency or data storage efficiency.

Transcript

1. SAKURA internet Inc.
   (C) Copyright 1996-2019 SAKURA internet Inc
   SAKURA Internet
   Research Center
   HeteroTSDB: An Extensible
   Time Series Database for Automatically Tiering
   on Heterogeneous Key-Value Stores
   July 16 2019 Yuuki Tsubouchi
   Yuuki Tsubouchi, Asato Wakisaka, Ken Hamada, Masayuki Matsuki,
   Hiroshi Abe and Ryosuke Matsumoto
   43rd IEEE International Conference Computer Software & Applications (COMPSAC)
   

   View Slide

2. 2
   1. Introduction
   2. Related Works
   3. Proposed Method
   4. Evaluation
   5. Conclusion
   Agenda
   

   View Slide

3. 1.
   Introduction
   

   View Slide

4. 4
   ɾDemand to internet services is increasing for availability and
   response speed.
   ɾA dedicated system for monitoring is necessary to identify a
   problem quickly in the systems.
   ɾHowever, it brings additional system setup and update work.
   ɾMany system administrators use a monitoring service to reduce
   these burdens.
   ɾA monitoring service builds a Time Series Database(TSDB) to store
   the time series data for monitoring.
   Background
   

   View Slide

5. 5
   ɾA monitoring system periodically collects metrics for measuring the
   state of systems.
   ɾrecords the metrics in a TSDB with high resolution
   ɾretains past data for a long time
   ɾThe challenges for TSDB
   ɾWrite processing efficiency
   ɾData storage efficiency
   The existing challenges for TSDB
   TSDB
   Heavy
   writing
   Long-term
   storage
   

   View Slide

6. 6
   ɾA monitoring service requires the addition of new functions.
   ɾto analyze long-term metrics for data warehouse
   ɾto label metrics by inverted index for flexible searching
   ɾIt is necessary for TSDB extensibility for data structure.
   ɾThe existing TSDBs configure a DBMS as a single building block, so
   that we should change the DBMS software itself to add a new data
   structure.
   ɾHowever, it is difficult to extend a DBMS with tightly coupled
   functions.
   The new challenge for TSDB
   

   View Slide

7. 7
   HeteroTSDB architecture
   ɾThe extensibility of the data structure
   ↪ Loosely-coupled architecture with heterogeneous DBMSs
   1. separate the single TSDB architecture into some parts such as log,
   index, time series data structure
   2. add a standalone DBMS per an additional data structure
   3. combine the parts as one loosely-coupled TSDB
   DBMS
   Single
   TSDB
   DBMS
   Index
   DBMS
   Log
   HeteroTSDB
   Time series data
   

   View Slide

8. 2.
   Related Works
   

   View Slide

9. 9
   What is a time series database?
   ɾA DBMS that has the following properties is called TSDB.
   (i) store a row of data that consists of timestamp, value, and optional
   tags(e.g., name)
   (ii) store multiple rows of time series data grouped together
   (iii) can query for rows of data
   (iv) can contain a timestamp or a time range in a query
   Quoted from A. Bader, O. Kopp, and F. Michael, “Survey and Comparison of Open Source Time Series Databases,” in
   Datenbanksysteme für Business, Technologie und Web (BTW) - Workshopband. GI, 2017, pp. 249–268.
   

   View Slide

10. 10
    The existing TSDBs
    ɾA TSDB with a general-purpose storage
    ɾIt can store data excepting time series data
    ɾe.g., RDBMS, NoSQL(Wide column store, Document store, Object
    storage)…
    ɾOpenTSDB[16]
    ɾA TSDB with a storage specialized for TSDB workload [17,10]
    ɾMaximizing processing and storage efficiency for TSDB workload
    ɾGorilla[17], InfluxDB[10]
    [16] OpenTSDB, http://opentsdb.net.
    [17] Pelkonen T et al., “Gorilla: A Fast, Scalable, In-Memory Time Series Database,” 41st International Conference on VLDB, vol.8, no.12, pp.1816–1827, 2015.
    [10] InfluxData, InfluxDB, https://www.influxdata.com/time-series-platform/influxdb/.
    

    

    View Slide

11. 11
    1. A service provider should change the DBMS software itself to add a
    new data structure.
    ɾIt is difficult to extend the DBMS with tightly coupled functions.
    2. A service provider are forced to learn, build and update TSDBs with a
    storage engine specialized for TSDB workload.
    ɾOpenTSDB that depends on timestamp sorting on HBase.
    ɾThe system administrators pay that operational costs.
    The problems with the existing methods
    

    View Slide

12. 3.
    Proposed Method
    

    View Slide

13. 13
    HeteroTSDB architecture
    1. The extensibility of the data structure
    ↪ Loosely-coupled architecture with heterogeneous DBMSs
    2. Reducing the operational costs of the DBMS specialized for TSDB
    ↪ Compatibility by key-value-based time series data structure
    3. Improving the efficiency of write processing and data storage
    ↪ Automatically tiering between in-memory KVS and on-disk KVS
    In-Memory
    KVS
    On—Disk
    KVS
    Tiering
    Single
    TSDB
    DBMS
    Index
    DBMS
    Log
    Our TSDB architecture
    

    View Slide

14. 14
    High-level architecture overview
    Message
    Broker
    Metric
    Writer
    In-Memory
    KVS
    Client
    On—Disk
    KVS
    Metric
    Mover
    (1) write (2) subscribe and write (3) migration
    Metric
    Reader
    (i) query
    (iii)
    merge data points
    (ii)
    read from each KVS
    (ii)
    1. How can we extend data structure?
    2. How do we model our time-series data structure as key-value format?
    3. How do we tier between an in-memory KVS and an on-disk KVS?
    Log
    

    View Slide

15. 1. How to extend data structure?
    

    View Slide

16. 16
    1. How to extend data structure?
    Message
    Broker
    Metric
    Writer
    In-Memory
    KVS
    Client
    On—Disk
    KVS
    DBMSs
    Metric
    Mover
    DBMSs
    Stream replication Batch replication
    ɾPros: The light writing workload to the
    additional DBMSs
    ɾCons: Lookup is delayed
    ɾPros: Immediate lookup
    ɾCons: The additional DBMSs are
    exposed to the heavy writing
    workload
    To guarantee consistency between the KVS and the additional DBMSs,
    their data structure should be idempotent.
    

    View Slide

17. 2. How to model the data structure as
    key-value?
    

    View Slide

18. 18
    2. How to model the data structure as key-value?
    name:wtimestamp
    Record1
    Key Value
    List of a pair of timestamp and value
    Record2
    1. Reading efficiently time series
    ɾstore multiple data points in the same series in one record.
    ɾdivide the series into fixed-width time windows.
    Time window
    name:wtimestamp List of a pair of timestamp and value
    

    View Slide

19. 19
    2. How to model the data structure as key-value?
    2. Range searching time series
    name:wtimestamp
    Key Value
    name:wtimestamp
    ɾAligning the timestamp to a multiple of the resolution before
    writing data points
    ɾ1482703011 (unix time) → 1482703020 (Multiple of 60 secs)
    Start
    End
    wtimestamp1
    wtimestamp2
    wtimestampN
    ɾ
    ɾ
    ɾ
    ɾ
    ……
    ……
    

    View Slide

20. 20
    2. How to model the data structure as key-value?
    name:wtimestamp
    Key Value
    Hash map of a pair of timestamp and value
    3. Preserving idempotent when rewriting the same datapoint
    ɾUse hash map
    { timestamp: value, …}
    name:wtimestamp
    

    View Slide

21. 3. How to automatically tier
    between an in-memory KVS and an
    on-disk KVS?
    

    View Slide

22. 22
    3. How to automatically tier between KVSs?
    1. The data points are written to the in-memory KVS.
    2. The old data points on the in-memory KVS is moved to the on-disk
    KVS in units of time series.
    ɾIf the in-memory KVS breaks down, it is recovered from write-ahead
    log(WAL).
    ɾA message broker implements WAL.
    ɾWe propose following two methods for data movement.
    ɾTimer method
    ɾCounting method
    In-Memory
    KVS
    On—Disk
    KVS
    Tiering
    Message
    broker
    Log
    

    View Slide

23. 23
    Data movement: Timer Method
    KVS KVS
    Record Record
    Trigger
    Record Trigger Record
    Metric
    Writer
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    ɾ
    1. A timer is set in units of record on the KVS.
    2. When the timer becomes 0, the KVS kicks a trigger.
    3. The trigger inserts the record to the another KVS.
    1.
    1.
    2.
    2.
    3.
    3.
    Timer (TTL)
    

    View Slide

24. 24
    Data movement: Counting Method
    Metric
    Writer
    In-Memory KVS
    1. Count data points when writing
    2. Read the batch of data points if the
    number is over a certain number
    3. Write the batch of data points
    On-Disk KVS
    At regular intervals
    Counting method is for KVS that doesn’t implement TTL
    

    View Slide

25. 25
    Summary of HeteroTSDB architecture
    1. Loosely-coupled architecture with heterogeneous DBMSs
    ɾStream replication and Batch replication
    2. Compatibility by key-value-based time series data structure
    ɾTime window with fixed width
    ɾIdempotent data structure by Hash map
    3. Automatically tiering between in-memory KVS and on-disk KVS
    ɾTimer method and Counting method
    In-Memory
    KVS
    On—Disk
    KVS
    Tiering
    Single
    TSDB
    DBMS
    Index
    DBMS
    Log
    Our TSDB architecture
    

    View Slide

26. 4.
    Evaluation
    

    View Slide

27. 27
    Evaluation
    1. Implementation
    2. Performance Evaluation
    ɾThe number of writes per minute
    ɾCPU and Memory resource usage
    3. Experience in production
    ɾPlease see the more details in our paper.
    

    View Slide

28. 1. Implementation
    

    View Slide

29. 29
    ɾUse the serverless platform on AWS
    ɾ3-tier: Memory - SSD - HDD
    Our implementation
    

    View Slide

30. 2. Performance Evaluation
    

    View Slide

31. 31
    3PMF *UFN 4QFDJpDBJUPO
    #FODINBSL
    $MJFOU
    &$JOTUBODF
    *OTUBODF5ZQF
    04
    ,FSOFM
    DYMBSHF
    "NB[PO-JOVY
    
    .FTTBHF
    #SPLFS
    ,JOFTJT%BUB4USFBNT
    /VNCFSPGTIBSET
    .FUSJD
    8SJUFS
    -BNCEB

    .FNPSZ
    3VOUJNF
    .#
    
    /PEFKT
    *O.FNPSZ
    ,74
    &MBTUJ$BDIFGPS3FEJT
    /PEFUZQF
    /VNCFSPGTIBSET
    DBDIFSMBSHF
    
    0O%JTL,74 %ZOBNP%# 8$6
    The benchmarker’s parameters
    • Number of agents: 40,000
    • Interval of data points
    transmission: 1 min
    • 100 data points / transmission
    We have observed changes for
    2 hours.
    

    View Slide

32. 32
    The number of writes per minute
    0
    1
    2
    3
    4
    5
    0 20 40 60 80 100 120
    datapoint writes / min (mega)
    minutes
    In-Memory KVS
    On-Disk KVS
    In-memory KVS: The number of
    writes per minute was constant at a
    value of about 4 M.
    On-disk KVS: The number of times of
    writing to the on-disk KVS took a value
    between 70k and 170k.
    1/20
    The number of writing per minute to
    the on-disk KVS is smaller to about
    1/20 than that to the in-memory KVS.
    

    View Slide

33. 33
    Resource usage of in-memory KVS
    0
    10
    20
    30
    40
    50
    60
    70
    80
    90
    100
    0 20 40 60 80 100 120
    0
    2
    4
    6
    8
    10
    12
    14
    16
    CPU usage (%)
    Free memory size (GB)
    minutes
    master CPU usage (%)
    slave1 CPU usage (%)
    slave2 CPU usage (%)
    Free memory size (GB)
    Free memory size transits
    while taking a constant
    value at around 10.5 GB.
    CPU usage transits while
    taking about 60~75%
    between the waveform
    shapes.
    

    View Slide

34. 34
    ɾThe number of writes per minute
    ɾWithout in-memory KVS, on-disk KVS should accept the same
    number of write times with the in-memory KVS.
    ɾHowever, HeteroTSDB reduces the number of writes to the on-
    disk KVS to about 1/20 of them directly to the on-disk KVS.
    ɾResource usage of in-memory KVS
    ɾThe memory usage on the in-memory KVS is expected to
    increase monotonically with time.
    ɾHowever, HeteroTSDB remains constant with the memory usage
    of the in-memory KVS.
    Consideration for performance
    

    View Slide

35. 3. Experience in production
    Deploying our architecture to the production of the monitoring
    service from August 2017 through August 2018.
    

    View Slide

36. 36
    Faults in production
    ɾ(1) The loss of data points on the in-memory KVS
    ɾThe data points that remained in the message broker were
    restored by running the MetricWriter.
    ɾ(2) The temporary write error by a software bug
    

    View Slide

37. 37
    Faults in production (1)
    ɾ(1) The loss of data points on the in-memory KVS
    ɾThe write load concentrated on a specific node of the in-memory
    KVS cluster.
    ɾThe memory consumption in the node reached the upper limit.
    ɾAs a result of the OS forcibly stopping the process, data of a
    plurality of specific metrics temporarily disappeared.
    ɾThe data points that remained in the message broker were
    restored by running the MetricWriter.
    

    View Slide

38. 38
    Faults in production (2)
    ɾ(2) The permanent write error by a software bug.
    ɾ Too many data points with the same metric name and the same
    timestamp were written in a short time.
    ɾThe write query size for the in-memory KVS exceeds the upper
    limit of implementation.
    ɾThe Lambda function returned errors, was continued to be retried,
    and processing of the entire MetricWriter was delayed.
    ɾWe modified the program of MetricWriter to eliminate duplication
    of data points before writing.
    

    View Slide

39. 5.
    Conclusion
    

    View Slide

40. 40
    Conclusion
    ɾWe have proposed the TSDB architecture focused on extensibility,
    compatibility, and efficiency.
    ɾWe have implemented HeteroTSDB on an AWS serverless platform
    to reduce the burden of construction of multiple DBMSs.
    ɾThe results of our experiment
    ɾHeteroTSDB reduces the number of writes to the on-disk KVS to
    about 1/20 and remains the constant memory
    ɾHeteroTSDB remains constant with the memory usage of the in-
    memory KVS.
    

    View Slide

41. 41
    Future Work
    ɾDemonstrating that the HeteroTSDB is practical in terms of
    performance after comparison with other TSDBs.
    ɾImplementing extensions of the data structure and evaluate the
    extensibility.
    ɾEvaluating the compatibility and show the limitation of
    HeteroTSDB.
    

    View Slide