대용량 지표 관리를 위한 TSDB 개발기

#TSCoke

무중단 확장이 가능한 prometheus 호환 Time series DBMS인 TSCoke에 대한 소개와, 성능 최적화 과정에 대해 소개합니다.

발표자 : yanoo.kim
카카오 분산기술셀에서 분산 스토리지 플랫폼을 개발하는 야누 입니다.

Transcript

1. ؀ਊ۝ ૑಴ ҙܻܳ ਤೠ
   TSDB ѐߊӝ
   ӣো਋ yanoo.kim
   ஠஠য়
   Copyright 2022. Kakao Corp. All rights reserved. Redistribution or public display is not permitted without written permission from Kakao.
   if(kakao)2022
   

   View Slide

2. ѐߊزӝ
   TSCoke ࣗѐ
   ࢿמ ୭੸ച
   അട ߂ ೱറ ҅ദ
   

   View Slide

3. ѐߊزӝ
   

   View Slide

4. ѐߊزӝ
   ؀ӏݽ ௿۞झఠܳ ҙܻೡٸ

   о੢ ݢ੷ ࠁח ചݶ?
   

   View Slide

5. 14:0015:0016:0017:0018:0019:0020:0021:00
   ѐߊزӝ
   ੉࢚૚റఐ૑
   2ਘ 3ਘ 4ਘ 5ਘ 6ਘ 7ਘ 8ਘ
   оਊҕр ࢎਊ۝
   0
   250
   500
   11ੌ 12ੌ 13ੌ 14ੌ 15ੌ 16ੌ 17ੌ
   ҃Ҋ য়ܨ
   ҳݒ҅ദࣻ݀ ࢲ࠺झಿ૕ഛੋ
   

   View Slide

6. טযաח ࢲߡ, טযաח ૑಴
   ૑಴੷੢ࣗ
   ૑಴੷੢ࣗ
   ૑಴੷੢ࣗ
   ૑಴੷੢ࣗ
   ૑಴੷੢ࣗ
   ૑಴੷੢ࣗ
   

   View Slide

7. “੉Ҕ ੷Ҕ ৈ۞ ૑಴ٜਸ
   ೞա੄ ࠭۽ ೤ச ࣻח হਸө?”
   

   View Slide

8. TSCoke ࣗѐ
   

   View Slide

9. TSCoke ࣗѐ
   TSDB + REDI
   COKE
   

   View Slide

10. TSDB
    द҅ৌ ؘ੉ఠ ୊ܻী ୭੸ചػ DBMS
    TSCoke ࣗѐ
    

    View Slide

11. Redicoke
    Redis ೐۽ష௒ਸ ࢎਊೞח

    NAND ӝ߈ ؀ਊ۝ ࠙࢑ Key-Value झషܻ૑
    TSCoke ࣗѐ
    

    View Slide

12. TSCoke ࣗѐ
    QD QD QD QD
    TSD TSD TSD TSD
    Redicoke
    REDO LOG
    SERIES META
    SAMPLE
    TSD ADDR INFO
    QD ADDR INFO
    ACCOUNT MAP
    SHARD MAP
    REPLICATION
    MGR
    ADMIN
    WATCHER
    TSCoke
    

    View Slide

13. WAL Replication & ޖ઺ױ ࣻಣഛ੢
    

    View Slide

14. Grafana Datasource োز Promql Compliance Test
    Prometheus ഐജࢿ
    

    View Slide

15. ࢿמ ୭੸ച
    

    View Slide

16. SampleChunk ୭੸ച
    Memory
    Local Storage
    Redicoke
    A B C
    6ਘ A:0 B:3 C:20
    A
    ঑୷
    A
    ױӝ ؘ੉ఠ(~30࠙)
    ઺ӝ ؘ੉ఠ(~3दр)
    ੢ӝ ؘ੉ఠ(3दр~)
    

    View Slide

17. SampleChunk ୭੸ച
    Memory
    Local Storage
    Redicoke
    A B C
    6ਘ A:0 B:3 C:20
    A
    ঑୷
    A
    ױӝ ؘ੉ఠ(~30࠙)
    ઺ӝ ؘ੉ఠ(~3दр)
    ੢ӝ ؘ੉ఠ(3दр~)
    ৘࢚ࠁ׮ ੣਷ LocalStorage I/O۽
    
    
    ੹୓ WriteThroughput੉ ઁডؽ
    

    View Slide

18. SampleChunk ୭੸ച
    - Local Stroage ো࢑ਸ ਤ೧ BadgerDBܳ ࢎਊ
    - LSMT(Log-Structured Merge Tree) ഋక
    - LSMTח য়ېػ ؘ੉ఠ੄ ઁѢ, Range Scanҗ੿ীࢲ੄ ബਯࢿ ૐ؀ܳ ਤ೧ Compactionҗ੿ਸ ࣻ೯ೣ
    - Compaction਷ CPU, IO ੗ਗਸ ௼ѱ ࣗݽೣ
    SSTable
    Mem table
    SSTable SSTable
    SSTable SSTable SSTable SSTable
    L1
    L2
    L3
    Compaction
    

    View Slide

19. SampleChunk ୭੸ച
    - Local Storageח 30࠙~3दр ࢎ੉੄ ઺ӝ ؘ੉ఠ ੷੢ࣗ. ٮۄࢲ য়ېػ ؘ੉ఠо হ਺ 

    -> Compaction ױࣽച оמ
    - Read, FullScan݅ ೙ਃೡࡺ, RangeScan਷ ೙ਃ হ਺.

    -> ੿۳ җ੿ ೙ਃ হ਺

    -> SSTable Merge ೙ਃ হ਺
    SSTable
    Mem table
    SSTable SSTable
    SSTable SSTable SSTable SSTable
    L1
    L2
    L3
    Compaction
    

    View Slide

20. SampleChunk ୭੸ച
    - LSS(Log-structured Segment Store)
    ਬബೠ ؘ੉ఠ
    ݅ܐػ ؘ੉ఠ
    Append
    Tail Block
    Truncate
    Block Block Block
    Head Block
    

    View Slide

21. SampleChunk ୭੸ച
    0
    900
    1800
    Badger LSS
    IOPS ݫݽܻ ࢎਊ۝
    CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz 

    SSD: Samsung - PM9A3 U.2
    ֫ਸ ࣻ۾ જ਺ ծਸ ࣻ۾ જ਺
    0
    250000
    500000
    Badger LSS
    

    View Slide

22. SeriesIndex ୭੸ച
    15.13GB
    9.23GB
    18.67GB SeriesIndex
    Others
    SampleChunk
    ೠ ֢٘੄ ݫݽܻ ࣗݽ۝
    

    View Slide

23. SeriesIndex ୭੸ച
    metric{method=“used”,instance=“192.168.2.101”}
    
    metric{method=“used”,instance=“192.168.2.117”}
    
    metric{method=“used”,instance=“192.168.2.163”}
    sum by (method)(rate(metric{method=“used”}[5m]))
    

    View Slide

24. SeriesIndex ୭੸ച
    
    metric,method,used = , ,
    
    metric,method,free = , ,
    
    metric,instance,192.168.2.101 =
    
    metric,instance,192.168.2.117 =
    
    metric,instance,192.168.2.163 =
    
    metric,instance,192.168.2.185 =
    
    metric,instance,192.168.2.214 =
    
    metric,instance,192.168.3.28 =
    
    Metric, LabelName, LabelValue Index
    sum by (method)(rate(metric{method=“used”}[5m]))
    

    View Slide

25. SeriesIndex ୭੸ച
    00 01 10 11
    00 01 10 11 00 01 10 11
    Radix Search
    Radix Tree
    00 11
    0000 0001 0010 1100 1110 1111
    Binary Search
    BTree
    

    View Slide

26. SeriesIndex ୭੸ച
    Dictionary Coding
    John,Male,Seoul
    
    Mary,Female,London
    
    Jane,Female,Seoul
    
    Peter,Male,Tokyo
    
    Sam,Male,London
    
    Emily,Female,Tokyo
    1,2,3
    
    4,5,6
    
    7,5,3
    
    8,2,9
    
    10,2,6
    
    11,5,9
    John=1
    
    Male=2
    
    Seoul=3
    
    Mary=4
    
    Female=5
    
    London=6
    
    Jane=7
    
    Peter=8
    
    Tokyo=9
    
    Sam=10
    
    Emily=11
    

    View Slide

27. SeriesIndex ୭੸ച
    metric,method,used = 1000,1001,1002
    
    metric,method,free = 1003,1004,1005
    
    metric,instance,192.168.2.101 = 1000
    
    metric,instance,192.168.2.117 = 1001
    
    metric,instance,192.168.2.163 = 1002
    
    metric,instance,192.168.2.185 = 1003
    
    metric,instance,192.168.2.214 = 1004
    
    metric,instance,192.168.3.28 = 1005
    
    Metric, LabelName, LabelValue Index
    Table
    metric=1
    
    method=2
    
    used=3
    
    free=4
    
    instance=5
    
    192.168.2.101=6
    
    192.168.2.117=7
    
    192.168.2.163=8
    
    192.168.2.185=9
    
    192.168.2.214=10
    
    192.168.3.28=11
    1,2,3 = 1000,1001,1002
    
    1,2,4 = 1003,1004,1005
    
    1,5,6 = 1000
    
    1,5,7 = 1001
    
    1,5,8 = 1002
    
    1,5,9 = 1003
    
    1,5,10 = 1004
    
    1,5,11 = 1005
    Metric, LabelName, LabelValue Index
    Dictionary Coding
    

    View Slide

28. SeriesIndex ୭੸ച
    A B
    A B A B
    A B A B A B A B
    Dictionary Coding
    0 1 2 3 4 5 6 7
    AAA
    
    AAB
    
    ABA
    
    ABB
    
    BAA
    
    BAB
    
    BBA
    
    BBB
    0=AAA
    
    1=AAB
    
    2=ABA
    
    3=ABB
    
    4=BAA
    
    5=BAB
    
    6=BBA
    
    7=BBB
    

    View Slide

29. SeriesIndex ୭੸ച
    0
    1
    2
    sync.M
    ap
    ART
    sync.M
    ap+D
    IC
    T
    ART+D
    IC
    T
    Index ઑഥ दр
    ծਸ ࣻ۾ જ਺
    0
    2500
    5000
    sync.M
    ap
    ART
    sync.M
    ap+D
    IC
    T
    ART+D
    IC
    T
    ݫݽܻ ࢎਊ۝
    ծਸ ࣻ۾ જ਺
    

    View Slide

30. ࠙࢑ ૕੄
    TBә ੷੢۝ = TB ઑഥ۝
    

    View Slide

31. ࠙࢑ ૕੄
    1,600ѐ
    4઱/15ୡ=161,280
    1,600*161,280 = 258,048,000 ѐ
    

    View Slide

32. ࠙࢑ ૕੄
    QD QD QD
    QD
    

    View Slide

33. ࠙࢑ ૕੄
    ਽׹दр(sec)
    ծਸ ࣻ۾ જ਺
    0
    15
    30
    1؀ 2؀ 3؀ 4؀ 5؀ 6؀ 7؀ 8؀
    CPU: Intel(R) Xeon(R) CPU E5-2620 v4 @ 2.10GHz 

    SSD: Samsung - PM9A3 U.2
    

    View Slide

34. അട ߂ ೱറ ҅ദ
    

    View Slide

35. ࢲ࠺झ അട
    2022/06/09 ࢎղ ࢲ࠺झ য়೑
    - ࢲ࠺झ҅੿ 126ѐ
    - series ѐࣻ 4রѐ
    - DKOS(ࢎղ Kubernates)ܳ ੉ਊ೧ ࢲ࠺झ ઺
    - TSCoke as a Service
    

    View Slide

36. ೱറ ҅ദ
    - Alarm, Recording Rule ӝמ
    - ৻ࠗ ࠙࢑ झషܻ૑ োز ӝמ
    - Down-sampling
    

    View Slide

37. хࢎ೤פ׮
    

    View Slide

38. ଵҊ ޙ೴
    1) Badger(github.com/dgraph-io/badger)
    2) Leis, Viktor, Alfons Kemper, and Thomas Neumann. “The adaptive radix tree: ARTful indexing for main-memory databases.” 2013 IEEE 29th International Conference on Data Engineering
    (ICDE). IEEE, 2013.
    

    View Slide