SAKURA internet Inc. (C) Copyright 1996-2019 SAKURA internet Inc SAKURA Internet Research Center HeteroTSDB: An Extensible Time Series Database for Automatically Tiering on Heterogeneous Key-Value Stores July 16 2019 Yuuki Tsubouchi Yuuki Tsubouchi, Asato Wakisaka, Ken Hamada, Masayuki Matsuki, Hiroshi Abe and Ryosuke Matsumoto 43rd IEEE International Conference Computer Software & Applications (COMPSAC)

2 1. Introduction 2. Related Works 3. Proposed Method 4. Evaluation 5. Conclusion Agenda

1. Introduction

4 ɾDemand to internet services is increasing for availability and response speed. ɾA dedicated system for monitoring is necessary to identify a problem quickly in the systems. ɾHowever, it brings additional system setup and update work. ɾMany system administrators use a monitoring service to reduce these burdens. ɾA monitoring service builds a Time Series Database(TSDB) to store the time series data for monitoring. Background

5 ɾA monitoring system periodically collects metrics for measuring the state of systems. ɾrecords the metrics in a TSDB with high resolution ɾretains past data for a long time ɾThe challenges for TSDB ɾWrite processing efficiency ɾData storage efficiency The existing challenges for TSDB TSDB Heavy writing Long-term storage

6 ɾA monitoring service requires the addition of new functions. ɾto analyze long-term metrics for data warehouse ɾto label metrics by inverted index for flexible searching ɾIt is necessary for TSDB extensibility for data structure. ɾThe existing TSDBs configure a DBMS as a single building block, so that we should change the DBMS software itself to add a new data structure. ɾHowever, it is difficult to extend a DBMS with tightly coupled functions. The new challenge for TSDB

7 HeteroTSDB architecture ɾThe extensibility of the data structure ↪ Loosely-coupled architecture with heterogeneous DBMSs 1. separate the single TSDB architecture into some parts such as log, index, time series data structure 2. add a standalone DBMS per an additional data structure 3. combine the parts as one loosely-coupled TSDB DBMS Single TSDB DBMS Index DBMS Log HeteroTSDB Time series data

2. Related Works

9 What is a time series database? ɾA DBMS that has the following properties is called TSDB. (i) store a row of data that consists of timestamp, value, and optional tags(e.g., name) (ii) store multiple rows of time series data grouped together (iii) can query for rows of data (iv) can contain a timestamp or a time range in a query Quoted from A. Bader, O. Kopp, and F. Michael, “Survey and Comparison of Open Source Time Series Databases,” in Datenbanksysteme für Business, Technologie und Web (BTW) - Workshopband. GI, 2017, pp. 249–268.

10 The existing TSDBs ɾA TSDB with a general-purpose storage ɾIt can store data excepting time series data ɾe.g., RDBMS, NoSQL(Wide column store, Document store, Object storage)… ɾOpenTSDB[16] ɾA TSDB with a storage specialized for TSDB workload [17,10] ɾMaximizing processing and storage efficiency for TSDB workload ɾGorilla[17], InfluxDB[10] [16] OpenTSDB, http://opentsdb.net. [17] Pelkonen T et al., “Gorilla: A Fast, Scalable, In-Memory Time Series Database,” 41st International Conference on VLDB, vol.8, no.12, pp.1816–1827, 2015. [10] InfluxData, InfluxDB, https://www.influxdata.com/time-series-platform/influxdb/. 


11 1. A service provider should change the DBMS software itself to add a new data structure. ɾIt is difficult to extend the DBMS with tightly coupled functions. 2. A service provider are forced to learn, build and update TSDBs with a storage engine specialized for TSDB workload. ɾOpenTSDB that depends on timestamp sorting on HBase. ɾThe system administrators pay that operational costs. The problems with the existing methods

3. Proposed Method

13 HeteroTSDB architecture 1. The extensibility of the data structure ↪ Loosely-coupled architecture with heterogeneous DBMSs 2. Reducing the operational costs of the DBMS specialized for TSDB ↪ Compatibility by key-value-based time series data structure 3. Improving the efficiency of write processing and data storage ↪ Automatically tiering between in-memory KVS and on-disk KVS In-Memory KVS On—Disk KVS Tiering Single TSDB DBMS Index DBMS Log Our TSDB architecture

14 High-level architecture overview Message Broker Metric Writer In-Memory KVS Client On—Disk KVS Metric Mover (1) write (2) subscribe and write (3) migration Metric Reader (i) query (iii) merge data points (ii) read from each KVS (ii) 1. How can we extend data structure? 2. How do we model our time-series data structure as key-value format? 3. How do we tier between an in-memory KVS and an on-disk KVS? Log

1. How to extend data structure?

16 1. How to extend data structure? Message Broker Metric Writer In-Memory KVS Client On—Disk KVS DBMSs Metric Mover DBMSs Stream replication Batch replication ɾPros: The light writing workload to the additional DBMSs ɾCons: Lookup is delayed ɾPros: Immediate lookup ɾCons: The additional DBMSs are exposed to the heavy writing workload To guarantee consistency between the KVS and the additional DBMSs, their data structure should be idempotent.

2. How to model the data structure as key-value?

18 2. How to model the data structure as key-value? name:wtimestamp Record1 Key Value List of a pair of timestamp and value Record2 1. Reading efficiently time series ɾstore multiple data points in the same series in one record. ɾdivide the series into fixed-width time windows. Time window name:wtimestamp List of a pair of timestamp and value

19 2. How to model the data structure as key-value? 2. Range searching time series name:wtimestamp Key Value name:wtimestamp ɾAligning the timestamp to a multiple of the resolution before writing data points ɾ1482703011 (unix time) → 1482703020 (Multiple of 60 secs) Start End wtimestamp1 wtimestamp2 wtimestampN ɾ ɾ ɾ ɾ …… ……

20 2. How to model the data structure as key-value? name:wtimestamp Key Value Hash map of a pair of timestamp and value 3. Preserving idempotent when rewriting the same datapoint ɾUse hash map { timestamp: value, …} name:wtimestamp

3. How to automatically tier between an in-memory KVS and an on-disk KVS?

22 3. How to automatically tier between KVSs? 1. The data points are written to the in-memory KVS. 2. The old data points on the in-memory KVS is moved to the on-disk KVS in units of time series. ɾIf the in-memory KVS breaks down, it is recovered from write-ahead log(WAL). ɾA message broker implements WAL. ɾWe propose following two methods for data movement. ɾTimer method ɾCounting method In-Memory KVS On—Disk KVS Tiering Message broker Log

23 Data movement: Timer Method KVS KVS Record Record Trigger Record Trigger Record Metric Writer ɾ ɾ ɾ ɾ ɾ ɾ 1. A timer is set in units of record on the KVS. 2. When the timer becomes 0, the KVS kicks a trigger. 3. The trigger inserts the record to the another KVS. 1. 1. 2. 2. 3. 3. Timer (TTL)

24 Data movement: Counting Method Metric Writer In-Memory KVS 1. Count data points when writing 2. Read the batch of data points if the number is over a certain number 3. Write the batch of data points On-Disk KVS At regular intervals Counting method is for KVS that doesn’t implement TTL

25 Summary of HeteroTSDB architecture 1. Loosely-coupled architecture with heterogeneous DBMSs ɾStream replication and Batch replication 2. Compatibility by key-value-based time series data structure ɾTime window with fixed width ɾIdempotent data structure by Hash map 3. Automatically tiering between in-memory KVS and on-disk KVS ɾTimer method and Counting method In-Memory KVS On—Disk KVS Tiering Single TSDB DBMS Index DBMS Log Our TSDB architecture

4. Evaluation

27 Evaluation 1. Implementation 2. Performance Evaluation ɾThe number of writes per minute ɾCPU and Memory resource usage 3. Experience in production ɾPlease see the more details in our paper.

1. Implementation

29 ɾUse the serverless platform on AWS ɾ3-tier: Memory - SSD - HDD Our implementation

2. Performance Evaluation

31 3PMF *UFN 4QFDJpDBJUPO #FODINBSL $MJFOU &$JOTUBODF *OTUBODF5ZQF 04 ,FSOFM DYMBSHF "NB[PO-JOVY .FTTBHF #SPLFS ,JOFTJT%BUB4USFBNT /VNCFSPGTIBSET .FUSJD 8SJUFS -BNCEB
 .FNPSZ 3VOUJNF .# /PEFKT *O.FNPSZ ,74 &MBTUJ$BDIFGPS3FEJT /PEFUZQF /VNCFSPGTIBSET DBDIFSMBSHF 0O%JTL,74 %ZOBNP%# 8$6 The benchmarker’s parameters • Number of agents: 40,000 • Interval of data points transmission: 1 min • 100 data points / transmission We have observed changes for 2 hours.

32 The number of writes per minute 0 1 2 3 4 5 0 20 40 60 80 100 120 datapoint writes / min (mega) minutes In-Memory KVS On-Disk KVS In-memory KVS: The number of writes per minute was constant at a value of about 4 M. On-disk KVS: The number of times of writing to the on-disk KVS took a value between 70k and 170k. 1/20 The number of writing per minute to the on-disk KVS is smaller to about 1/20 than that to the in-memory KVS.

33 Resource usage of in-memory KVS 0 10 20 30 40 50 60 70 80 90 100 0 20 40 60 80 100 120 0 2 4 6 8 10 12 14 16 CPU usage (%) Free memory size (GB) minutes master CPU usage (%) slave1 CPU usage (%) slave2 CPU usage (%) Free memory size (GB) Free memory size transits while taking a constant value at around 10.5 GB. CPU usage transits while taking about 60~75% between the waveform shapes.

34 ɾThe number of writes per minute ɾWithout in-memory KVS, on-disk KVS should accept the same number of write times with the in-memory KVS. ɾHowever, HeteroTSDB reduces the number of writes to the on- disk KVS to about 1/20 of them directly to the on-disk KVS. ɾResource usage of in-memory KVS ɾThe memory usage on the in-memory KVS is expected to increase monotonically with time. ɾHowever, HeteroTSDB remains constant with the memory usage of the in-memory KVS. Consideration for performance

3. Experience in production Deploying our architecture to the production of the monitoring service from August 2017 through August 2018.

36 Faults in production ɾ(1) The loss of data points on the in-memory KVS ɾThe data points that remained in the message broker were restored by running the MetricWriter. ɾ(2) The temporary write error by a software bug

37 Faults in production (1) ɾ(1) The loss of data points on the in-memory KVS ɾThe write load concentrated on a specific node of the in-memory KVS cluster. ɾThe memory consumption in the node reached the upper limit. ɾAs a result of the OS forcibly stopping the process, data of a plurality of specific metrics temporarily disappeared. ɾThe data points that remained in the message broker were restored by running the MetricWriter.

38 Faults in production (2) ɾ(2) The permanent write error by a software bug. ɾ Too many data points with the same metric name and the same timestamp were written in a short time. ɾThe write query size for the in-memory KVS exceeds the upper limit of implementation. ɾThe Lambda function returned errors, was continued to be retried, and processing of the entire MetricWriter was delayed. ɾWe modified the program of MetricWriter to eliminate duplication of data points before writing.

5. Conclusion

40 Conclusion ɾWe have proposed the TSDB architecture focused on extensibility, compatibility, and efficiency. ɾWe have implemented HeteroTSDB on an AWS serverless platform to reduce the burden of construction of multiple DBMSs. ɾThe results of our experiment ɾHeteroTSDB reduces the number of writes to the on-disk KVS to about 1/20 and remains the constant memory ɾHeteroTSDB remains constant with the memory usage of the in- memory KVS.

41 Future Work ɾDemonstrating that the HeteroTSDB is practical in terms of performance after comparison with other TSDBs. ɾImplementing extensions of the data structure and evaluate the extensibility. ɾEvaluating the compatibility and show the limitation of HeteroTSDB.