Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

CloudNative Days Tokyo 2021での登壇資料です
https://event.cloudnativedays.jp/cndt2021/talks/1252

A3966f193f4bef226a0d3e3c1f728d7f?s=128

LINE Developers
PRO

November 04, 2021
Tweet

Transcript

  1. 1 Grafana LokiͰߏங͢Δ େن໛ϩάϞχλϦϯάج൫ CNDT2021 LINEגࣜձࣾ Hiroki Sakamoto / @taisho6339

  2. 2 ࣗݾ঺հ - Title: Senior Software Engineer@LINE Corp - Role:

    Private Cloud։ൃ૊৫ͷSRE - Mission Private CloudΛԣஅͨ͠৴པվળ - Interest: Kubernetes, ෼ࢄγεςϜ, ےτϨ, OSS׆ಈ - Twitter: @taisho6339
  3. like Prometheus but for logs • ϩάͷอଘͱݕࡧػೳ • ϩάϕʔεͷΞϥʔςΟϯάػೳ •

    ϩάϕʔεͷϝτϦΫε࡞੒ػೳ • ϚϧνςφϯτͷDefault Support 3 Lokiͱ͸Կ͔ʁ
  4. 4 Lokiͱ͸Կ͔ʁ

  5. 5 Lokiͱ͸Կ͔ʁ

  6. 6 ҆͘େ༰ྔͷϩάΛอଘՄೳ

  7. 7 Private Cloud “Verda” in LINE is based on OpenStack.

    since 2016~ FaaS PaaS IaaS NAT LB Bare metal
  8. 8 Private Cloud “Verda” in LINE Virtual Machines 74000+ 30000+

    4000+ Physical Machines Hypervisors
  9. 9 20 TB / day application logs

  10. 10 Loki is suitable for us!

  11. 11 Lokiͷ೉͠͞ Loki͸ϚΠΫϩαʔϏε • ֤ίϯϙʔωϯτɺ֤Ωϟογϡͷ࢓૊Έͱ໾ׂ͕ෆ໌ྎ • ϩάσʔλ͸Ͳ͜ͰͲΜͳܗࣜͰͲͷ͘Β͍อ࣋͞ΕΔ͔ෆ໌ྎ • ετϨʔδো֐࣌͸Ͳ͏͍͏ڍಈʹͳΔͷ͔ෆ໌ྎ •

    ຊ൪Ͱӡ༻͢ΔͳΒԿΛߟྀ͠ͳ͍ͱ͍͚ͳ͍ͷ͔͕ෆ໌ྎ
  12. 12 ຊηογϣϯͷΰʔϧ ࠃ಺Ͱ࠷΋ৄࡉʹղઆ͢Δ͜ͱΛ໨ࢦ͠·͢ • ମܥతʹLokiͷίϯϙʔωϯτͷ໾ׂͱ࢓૊ΈΛ஌Δ • τϥϒϧ࣌ʹ΋ݪҼͷಛఆ͕ਝ଎ʹͰ͖ΔΑ͏ʹͳΔ • ࣗྗͰΩϟύγςΟ؅ཧɺύϑΥʔϚϯενϡʔχϯάͰ͖ΔΑ ͏ʹͳΔ

  13. 13 ຊηογϣϯ૝ఆߏ੒ Loki Version: v2.3.0 Cache: Memcached Chunk Storage: AWS

    S3 Index Storage: AWS S3 + BoltDB Shipper
  14. 14 1) ϩάͷॻ͖ࠐΈϓϩηεΛ஌Δ 2) ϩάͷಡΈࠐΈϓϩηεΛ஌Δ 3) ো֐࣌ͷڍಈΛ஌Δ ຊηογϣϯͷRoadmap

  15. 15 1) ϩάͷॻ͖ࠐΈϓϩηε

  16. 16 Overview

  17. 17 ॻ͖ࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Loki Clients Storage Distributor

    Ingesters Clients (Promtail, Fluentd)
  18. 18 ॻ͖ࠐΈϓϩηε Overview Amazon S3 Chunk Cache Index Write Cache

    Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) • Fluentd΍PromtailͳͲͷɺϩάૹ৴Client • ϩάΛಡΈऔΓLokiͷΤϯυϙΠϯτ΁ૹ৴͢Δ
  19. 19 Amazon S3 Chunk Cache Index Write Cache Loki Clients

    Storage Distributor Ingesters Clients (Promtail, Fluentd) • ϦΫΤετͷόϦσʔγϣϯΛߦ͏ • ద੾ͳIngester΁ϧʔςΟϯά͢Δ ॻ͖ࠐΈϓϩηε Overview
  20. 20 Amazon S3 Chunk Cache Index Write Cache Loki Clients

    Storage Distributor Ingesters Clients (Promtail, Fluentd) ϩάΛ࣮ࡍʹετϨʔδʹอଘ͢Δ Ұఆ࣌ؒόοϑΝϦϯά͔ͯ͠Βετ Ϩʔδʹอଘ ॻ͖ࠐΈϓϩηε Overview
  21. 21 Amazon S3 Loki Clients Storage Distributor Ingesters Clients (Promtail,

    Fluentd) • S3ʹϩάΛӬଓԽ • MemcachedʹϩάͷΩϟογϡΛอଘ Chunk Cache ॻ͖ࠐΈϓϩηε Overview
  22. 22 Client͔ΒDistributor΁ͷૹ৴

  23. 23 Client -> Distributor Clients (Promtail, Fluentd) Distributor HTTP Headers

    X-Scope-OrgID : <Tenant ID> TenantIDΛRequest Headerʹهࡌ
  24. 24 Client -> Distributor Clients (Promtail, Fluentd) Distributor {service=“keystone”, hostname=“host1”}

    00:00:02 keystone log body {service=“keystone”, hostname=“host1”} 00:00:03 keystone log body {service=“keystone”, hostname=“host1”} 00:00:04 keystone log body Loki΁ૹΔϩάσʔλߏ଄
  25. 25 Client -> Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body

    Stream Log Body TS
  26. 26 Client -> Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body

    ϩά͸͍͔ͭ͘ͷϥϕϧΛ࣋ͭɻ TenantIDͱϥϕϧͷ૊Έ߹Θͤͷ ҰͭҰͭΛɺʮStreamʯͱݺͿ Stream Log Body TS
  27. 27 DistributorͰͷόϦσʔγϣϯ • ϥϕϧͷܗࣜ͸ਖ਼͍͔͠ʁ • Rate limitΛ௒͑ͳ͍͔ʁ Clients (Promtail, Fluentd)

    Distributor
  28. 28 Distributor DistributorͰͷόϦσʔγϣϯ Distributor Distributor Distributor Distributorಉ࢜ͰΫϥελϦϯά • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ •

    ingestion_rate_strategy͕global ͳΒΫϥελશମͰingestion rateΛ੍ޚ͢Δ
  29. 29 Distributor DistributorͰͷόϦσʔγϣϯ Distributor Distributor Distributor DistributorશମͰόϦσʔγϣϯ ͢ΔͨΊͷΫϥελϦϯά

  30. 30 Distributor͔ΒIngester΁ͷૹ৴

  31. 31 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ݕࡧͷͨΊʹ(ޙड़)

    ෳ਺ͷIngesterʹ ৑௕Խͯ͠ϩάΛૹΔ
  32. 32 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor Ingesterಉ࢜ͰΫϥελϦϯά

    • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ • Consistent HashͷRingʹͳͬͯ ͍Δ
  33. 33 Distributor -> Ingesters Ingester Ingesters Ingester Ingester FNV1-32bitͰHash஋Λੜ੒ tenantID

    + {service=“keystone”, hostname=“host1”} a6965cd7
  34. 34 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor a6965cd7

    Consistent Hashʹج͍ͮͯɺ ࢉग़ͨ͠Hash஋ʹରԠ͢ΔIngester Λreplication factor෼ཁٻ
  35. 35 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ؼ͖ͬͯͨෳ਺ͷIngesterʹɺ

    ಉ͡ϩάΛಉ࣌ૹ৴
  36. 36 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor OK

    OK Fail
  37. 37 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ա൒਺͕OKͳΒ੒ޭ

    OK OK Fail
  38. 38 IngesterͷRequest Handling

  39. 39 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 {service=“keystone”, hostname=“host1”} Ingester

  40. 40 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks StreamͰGrouping͞Εɺ

    Chunkͱ͍͏ܗࣜͰAppend Ingester
  41. 41 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL

    Segment Ingester WALʹه࿥
  42. 42 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL

    Segment Ingester OK
  43. 43 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL

    Segment Ingester ΋͠Stream಺Ͱ࠷ޙʹड͚औͬͨϩάͷ࣌ؒΑΓ લͷ࣌ؒͷϩά͕དྷͨ৔߹͸ϦΫΤετΛࣦഊͤ͞Δ Out of order entry error
  44. 44 ChunkόοϑΝϦϯά

  45. 45 IngesterͷChunkόοϑΝϦϯά • Ұఆ਺ͷϩάΛStream͝ͱʹ ʮChunkʯʹ·ͱΊΔ • Chunk͸ϝϞϦ্ʹอଘ͞ΕΔ • HeadɺBlocksͱ͍͏഑ྻΛอ࣋ Head

    Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk
  46. 46 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk {service=“keystone”, hostname=“host1”}
  47. 47 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk Log Append ड͚औͬͨϩά͸Ұ୴Head΁ Append͞ΕΔ
  48. 48 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢
  49. 49 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk Log Log Log Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢
  50. 50 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Blockʹ௥Ճ ҰఆαΠζཷ·ͬͨΒ ઃఆͨ͠ܗࣜͰѹॖ͠ɺ blocksʹ௥Ճ͢Δ
  51. 51 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Block͕Ұఆ਺ཷ·ͬͨΒ Read Only ModeʹͳΓɺFlush Queue΁ (Default 10 blocks, target chunk sizeͰࢦఆ)
  52. 52 Ingester͔ΒStorage΁Flush

  53. 53 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Chunk Cache Disk సஔIndex
  54. 54 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε ৚݅Λຬͨ͢ChunkΛݕ஌ • ࢦఆαΠζʹ౸ୡ • ࠷ޙͷߋ৽͔Βchunk idle periodܦա • max_chunk_ageܦա
  55. 55 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Disk సஔΠϯσοΫε Flush Queue΁Enqueue
  56. 56 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Disk Enqueue͞Εͨ΋ͷΛFlush 1. ChunkΛS3΁อଘ 2. Chunk Cache΁อଘ(Write Through) 3. సஔIndexΛϩʔΧϧBoltDBʹอଘ Chunk Cache సஔIndex
  57. 57 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Goroutine Goroutine Amazon S3 Disk Ingester΁ͷϦΫΤετ͸ ෳ੡͞Ε͍ͯΔͷͰɺ طʹCacheʹೖ͍ͬͯΔChunk͸ Storage΁ͷॻ͖ࠐΈ͕ൃੜ͠ͳ ͍Α͏ʹ੍ޚ͍ͯ͠Δ Chunk Chunk Cache సஔIndex
  58. 58 Cacheͷෛՙ෼ࢄ Chunk Cache Ingesters Chunk Cache Chunk Cache Chunk1

    Key Chunk2 Key Chunk3 Key ઃఆʹΑͬͯConsistent HashʹΑΓɺ ChunkͷKeyΛݩʹ෼ࢄͯ͠อଘͰ͖Δ
  59. 59 Write Ahead Log

  60. 60 ChunkͷFlush(࠶ܝ) Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε
  61. 61 IngesterͷChunkͷ࣋ͪํ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Flush͞ΕΔલʹϓϩηε͕ࢭ·Δͱ Chunk͕شൃ͢Δ
  62. 62 WALͷҙٛ ӬଓԽͯ͠Memoryͷشൃʹඋ͑Δ

  63. 63 LokiͷWALͷಛ௃ • ϩάड৴࣌ʹɺMemoryͱWAL྆ํʹॻ͖ࠐΉ • WALॻ͖ࠐΈ͕ࣦഊͯ͠΋ॲཧΛࣦഊͤ͞ͳ͍ • Ingesterͷϓϩηεىಈ࣌ʹWAL͔Βͷ෮چॲཧ͕ೖΔ • Ұఆظؒ͝ͱʹෆཁͳWAL͸ύʔδ͞ΕΔ

  64. 64 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Log Entry ϩάΛૹ৴
  65. 65 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Append SegmentϑΝΠϧʹ rawσʔλͷ··௥ه͞ΕΔ
  66. 66 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Create! 1ϑΝΠϧ͕େ͖͘ͳͬͯ͘Δͱ ผͷSegmentϑΝΠϧΛ࡞੒ Segment2
  67. 67 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Goroutine Disk Segment1 Segment2 Ұఆظؒ͝ͱʹ ෆཁͳSegmentϑΝΠϧͷ ύʔδॲཧ͕ೖΔ
  68. 68 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Segment2 Segment3 Create! SegmentΛҰͭਐΊΔ
  69. 69 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 IngesterͷະFlush ChunkΛ Checkpointͱݺ͹ΕΔ εφοϓγϣοτͱͯ͠อଘ
  70. 70 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 MemoryChunkߏ଄ͷ··อଘ͢Δ ͷͰblock͸ѹॖܗࣜɺ headঢ়ଶͷϩά͸ඇѹॖͱͳΔ
  71. 71 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment3 Checkpoint1 ࠷৽ͷCheckpointͱSegmentΛ࢒ ͯ͢͠΂ͯ࡟আ͢Δ
  72. 72 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment3 Checkpoint1 ͦͷ࣌఺ͰFlush͞Ε͍ͯͳ͍ ͢΂ͯͷϩάؚ͕·Ε͍ͯΔ
  73. 73 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ
  74. 74 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Stream1 Stream2 Stream1 Stream2 Memory಺ʹ෮ݩ͢Δ
  75. 75 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Stream1 Stream2 Stream1 Stream2 ෮ݩ͕׬ྃ͢Δ·Ͱ͸ϓϩηεΛىಈ͠ͳ͍
  76. 76 Ingester্ͷσʔλͱEncodeܗࣜ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Disk Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷
  77. 77 2) ϩάͷಡΈࠐΈϓϩηε

  78. 78 Overview

  79. 79 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend Queriers Query Result Cache Ingesters
  80. 80 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ϦΫΤετΛड෇͚Δ
  81. 81 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ड͚औͬͨΫΤϦΛ࣌ؒͳͲͰ ෼ׂͯ͠ΩϡʔΠϯά͢Δ
  82. 82 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ෳ਺ͷQuerier͕Ωϡʔ͔ΒQuery Λड͚औΓϋϯυϦϯά͢Δ Queriers
  83. 83 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ͢΂ͯͷIngesterʹରͯ͠ɺ MemoryChunk͔ΒQueryʹMatch ͢Δ΋ͷΛཁٻ Queriers Ingesters
  84. 84 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend QueryʹରԠ͢ΔసஔIndexΛऔಘ͢Δ ͜ͷͱ͖CacheʹಁաతʹΞΫηε Queriers
  85. 85 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend Cacheʹଘࡏ͠ͳ͍৔߹͸ɺ ϩʔΧϧͷBoltDB͔ΒMatch͢ΔIndexΛऔಘ ͦͯ݁͠ՌΛCacheʹอଘ͢Δ(snappy) Queriers BoltDB
  86. 86 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend సஔIndex͔Βର৅ͷChunkΛׂΓग़͢ Queriers
  87. 87 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ChunkΛऔಘ͢Δ Cacheʹͳ͍΋ͷ͸Storage͔Βऔಘ͠ɺ Cacheʹอଘ͢Δ Queriers
  88. 88 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ͢΂ͯͷQuerier͔Βͷ݁ՌΛड͚औΓɺ ू໿ɺιʔτɺॏෳഉআΛ࣮ࢪ Queriers
  89. 89 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ݁ՌΛQuery Result Cache΁อଘ͢Δ Queriers Query Result Cache
  90. 90 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend Ϩεϙϯεฦ٫ Queriers Query Result Cache
  91. 91 సஔIndex͔Βର৅Chunkͷબఆ

  92. 92 సஔIndexͷ໨త S3͔ΒChunkΛ࠷খ࿑ྗͰऔಘ͢Δ͜ͱ

  93. 93 సஔIndex͔Βର৅ChunkͷߜΓࠐΈ 1.LabelͷKeyͱValueͷ૊Έ߹Θ͔ͤΒStreamͷID (Series ID)Λऔಘ͢Δ 2.Series IDͱ࣌ؒൣғ͔ΒChunkͷKeyΛऔಘ͢Δ 3.ChunkͷKey͔ΒS3্ɺMemcached্ͷύεΛׂ Γग़͠ɺ௚઀ChunkΛDownload

  94. 94 SeriesID {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body key, valueͷ૊Έ߹Θͤͷsha256

    9ac2adda49e899b312a9abb895656b1ab26c9858fd500f2ae3983d5309b39363/ ϩά SeriesID
  95. 95 Chunk Key {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body Tenant1/a6965cd7:Chunk։࢝࣌ؒ:Chunkऴྃ࣌ؒ

    key, valueͷ૊Έ߹ΘͤͷHash஋ a6965cd7 ϩά FingerPrint Chunk Key
  96. Hash Value Range Value Value TenantID + LabelName Hash(Label Value)

    + SeriesID Label Value సஔIndexͷߏ଄Πϝʔδ Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index
  97. Hash Value Range Value Value TenantID + LabelName Hash(Label Value)

    + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index Hash + RangeͰϢχʔΫߦΛಛఆ͢Δ సஔIndexͷߏ଄Πϝʔδ
  98. Hash Value Range Value Value TenantID + LabelName Hash(Label Value)

    + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index సஔIndexͷߏ଄Πϝʔδ Range Value͸ൣғݕࡧɺιʔτʹར༻Ͱ͖Δ
  99. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value)

    + SeriesID) Value (Label Value) Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler ςʔϒϧΠϝʔδ సஔIndexͷߏ଄Πϝʔδ
  100. ͜ͷIndex͸ LabelͷKeyͱValueͷύλʔϯ෼͚ͩ࡞ΒΕΔ ΧʔσΟφϦςΟͷߴ͍ϥϕϧ͸ ͜ͷIndex͕େྔʹ࡞ΒΕΔ͜ͱʹͳΔ సஔIndexͷߏ଄Πϝʔδ

  101. SeriesID͔ΒChunk KeyΛҾ͘Index సஔIndexͷߏ଄Πϝʔδ Hash Value Range Value Value TenantID +

    SeriesID Chunkͷ։࢝࣌ؒ + Chunk Key nil
  102. 102 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} |= “level=ERROR” LogQL ϥϕϧϚον෦ ϑΟϧλ෦

  103. 103 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} |= “level=ERROR” LogQL ϥϕϧϚον෦ ϑΟϧλ෦

    IndexΛ࢖͏ͷ͸͜ͷ෦෼ͷධՁ
  104. 104 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} ෼ׂ

  105. 105 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} ͦΕͧΕͰϚονϯά৚݅ ͔ΒసஔIndexΛऔಘ సஔIndex

    సஔIndex
  106. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value)

    + SeriesID) Value Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”} ώοτ͢Δͷ͸͜ͷϨίʔυ
  107. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value)

    + SeriesID) Value Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”} ର৅SeriesID
  108. 108 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} Index͔ΒSeriesIDΛநग़ ྆ํʹڞ௨͢Δ΋ͷ͚ͩ࠾༻ SeriesIDs

    సஔIndex సஔIndex
  109. 109 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} SeriesIDͱ࣌ؒൣғ͔Β Chunk KeyΛநग़

    SeriesIDs సஔIndex Chunk Keys సஔIndex సஔIndex
  110. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min Hash Value (TenantID

    + SeriesID) Range Value (Chunk։࢝࣌ؒ + Chunk Key) Value Tenant1:c79abadeff 1635252768:chunk1 nil Tenant1:c79abadeff 1635256368:chunk2 nil Tenant1:c79abadeff 1635260768:chunk3 nil Index͔ΒͷChunk KeyׂΓग़͠
  111. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min Hash Value (TenantID

    + SeriesID) Range Value (Chunk։࢝࣌ؒ + Chunk Key) Value Tenant1:c79abadeff 1635252768:chunk1 nil Tenant1:c79abadeff 1635256368:chunk2 nil Tenant1:c79abadeff 1635260768:chunk3 nil Index͔ΒͷChunk KeyׂΓग़͠ ର৅ChunkͷRecord
  112. 112 Ϩεϙϯεͷੜ੒ͱฦ٫

  113. 113 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks ॳظঢ়ଶͰ͸࣮ମΛ࣋ͨͣɺ ಡΈࠐΈ໋ྩ͕͞ΕͨλΠϛϯάͰετϨʔδ(Ωϟογϡ) ʹChunkΛऔΓʹߦ͘Lazy ChunkΛੜ੒

  114. 114 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Ingester͔ΒͷChunkΛ͋Θͤͯɺ IteratorΛੜ੒

  115. 115 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Ϩεϙϯε͕نఆ݅਺ʹୡ͢Δ·Ͱɺ

    IteratorΛಡΈࠐΜͰ͍͘ ϩάͷFilter৚݅͸͜͜ͰධՁ͞ΕΔ |= “level=ERROR”
  116. 116 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response LazyChunkΛಡΈࠐΉ৔߹͸ɺ

    Ωϟογϡ΁ɺͳ͚Ε͹Storage΁ ChunkΛ໰͍߹ΘͤΔ Amazon S3 Chunk Cache
  117. 117 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Storage͔ΒಡΈࠐΜͩ৔߹͸ɺ

    औಘޙʹCacheʹอଘ͢Δ Amazon S3 Chunk Cache
  118. 118 Query Sharding

  119. 119 Queryͷ෼ׂઓུ 1. ࣌ؒ͝ͱʹ෼ׂ͢Δ • 1࣌ؒ෼ͷϩάΛݕࡧ͢Δ৔߹ɺ15෼Ͱ෼ׂ͢ΔઃఆͳΒ4 ͭͷΫΤϦʹ෼ղ͞Ε࣮ͯߦ͞ΕΔ 2. సஔIndexΛSharding͢Δ •

    ͋Β͔͡ΊసஔIndexʹShard൪߸Λ͍Ε͓͖ͯɺ QueryFrontend͕QueryΛ෼ׂ͠ɺͦΕͧΕʹShard൪߸Λ ૠೖͯ͠QuerierʹΘͨ͢
  120. Hash Value Range Value Value TenantID + LabelName Shard Number

    + Hash(Label Value) + SeriesID Label Value సஔIndex΁ͷShard൪߸ຒΊࠐΈ Shard Number = SeriesID % shard count
  121. 121 సஔIndex΁ͷShard൪߸ຒΊࠐΈ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 Shard

    Number Stream
  122. 122 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”} |= “level=ERROR” LogQL Querier for shard 1

    Querier for shard 12 Querier for shard 16 QueryΛShardͰ෼ׂ
  123. 123 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”}

    |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys औΕΔChunk͕shardͰ෼ׂ͞ΕΔ
  124. 124 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”}

    |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys |= “level=ERROR”ͷfilterॲཧΛ෼ׂॲཧͰ͖Δ
  125. 125 BoltDB ShipperʹΑΔIndex؅ཧ

  126. 126 BoltDB Shipper Ingester Shipper Disk Querier Shipper Disk Index

    1 Index 2 Index 1 Index 2 Amazon S3
  127. 127 BoltDB Shipper - Ingester side Ingester Shipper Disk Querier

    Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 Ұఆ࣌ؒ͝ͱʹϩʔΧϧDiskʹ͋ΔIndex ΛS3΁Ξοϓϩʔυ͢Δ Ξοϓϩʔυޙʹ࡟আ͢Δ
  128. 128 BoltDB Shipper - Querier side Ingester Shipper Disk Querier

    Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 • ىಈ࣌ʹS3ʹ͋ΔIndexΛDownload • Query࣌ʹ଍Γͳ͍Index͸S3͔Β౎౓ μ΢ϯϩʔυ • Ұఆ࣌ؒ͝ͱʹ࠷ऴ࢖༻͔Β CacheTTLܦաͨ͠IndexΛ࡟আ
  129. 129 BoltDB Shipper Ingester Shipper Disk Querier Shipper Disk Index

    1 Index 2 Index 1 Index 2 Amazon S3 Ingester, Querier͸ϩʔΧϧͰIndexΛѻ͍ɺ Shipper͕ඇಉظͰIndexΛStorageͱಉظ͢Δ
  130. 130 ֤ίϯϙʔωϯτͷ໾ׂ·ͱΊ

  131. 131 Name ໾ׂ λΠϓ σʔλͷ࣋ͪԽ ΫϥελϦϯά༗ແ Distributor όϦσʔγϣϯͱIngester΁ͷϧʔςΟϯά Stateless ༗

    Ingester σʔλͷόοϑΝϦϯάͱFlush Stateful Memory: Chunks(raw + ѹॖ) Disk: WAL(raw + ѹॖ) సஔIndex(ѹॖ) ༗ Query Frontend ΫΤϦͷ෼ׂɺΩϡʔ੍ޚ Stateless ແ Querier ΫΤϦͷ࣮ߦ Stateful Disk: సஔIndexͷCache(ѹॖ) ແ Chunk Cache ChunkͷΩϟογϡ Stateful Memory: Chunks(ѹॖ) ༗(ΫϥΠΞϯταΠυ) Index Read Cache IndexͷRead༻Ωϟογϡ Stateful Memory: సஔIndex(Snappy) ༗(ΫϥΠΞϯταΠυ) Index Write Cache ಉ͡Indexͷॻ͖ࠐΈ͕ෳ਺ൃੜ͠ͳ͍Α͏ʹ ͢ΔͨΊͷ੍ޚ༻Ωϟογϡ (BoltDB ShipperͰ͸ෆཁ) Stateful Memory: Chunk Key(raw) ༗(ΫϥΠΞϯταΠυ) Query Result Cache ΫΤϦͷ݁ՌͷΩϟογϡ Stateful Memory: Query Result(raw) ༗(ΫϥΠΞϯταΠυ)
  132. 132 3) ো֐࣌ͷڍಈΛ஌Δ

  133. 133 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ

  134. 134 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ S3ো֐࣌ʹඋ͑Δ

  135. 135 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk Segment1 Checkpoint1 ͕ࣦ͜͜ഊ సஔIndex
  136. 136 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 Flush͸ඇಉظͳͷͰ ॻ͖ࠐΈࣗମ͸ࣦഊ͠ͳ͍
  137. 137 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ MemoryͱDiskʹͨ·Γଓ͚Δ
  138. 138 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ ઌʹMemory͕͋;ΕɺOOM
  139. 139 WALͷ࢓૊Έ(࠶ܝ) Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ
  140. 140 WALͷ࢓૊Έ(࠶ܝ) Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Stream1 Stream2 Stream1 Stream2 Memory಺ʹ෮ݩ͢Δ
  141. 141 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 WAL͸࣮࣭MemoryͷSnapshot ϩʔυ੒ޭͯ͠΋͍ۙ͏ͪʹOOM ϩʔυࣦഊͨ͠Βͦ΋ͦ΋ىಈ͠ͳ͍
  142. 142 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ϘτϧωοΫ͸IngesterͷMemory
  143. 143 Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Disk

    Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷ Ingester্ͷσʔλͱEncodeܗࣜ(࠶ܝ)
  144. 144 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ ରࡦ1. ଱͍͑ͨ࣌ؒ෼ͷMemoryΛੵΉ • ࣌ؒ͋ͨΓͷϩάྔ / ѹॖൺ * ࣌ؒ

    * replication factor ※ѹॖൺ͸ฐ؀ڥͰ͸gzipѹॖͰ10~18ഒͷѹॖൺ
  145. 145 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ 1೔ͷྲྀྔ10TBͷ؀ڥͰ1࣌ؒ଱͑Δ(ฐࣾͷ1Ϧʔδϣϯ) 1000 / 24 = 41.6 GB /

    hour • Replication Factor = 1 • Ingester * 11୆ • Memory: 4GiB • Disk: 8GiB(ϚʔδϯΛऔͬͯMemoryͷ2ഒ) • Chunk Cache * 14୆ • Memory: 3GiB
  146. 146 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Replication Factor͸1Ͱ͍͍ͷ͔ʁ • Flush͞ΕΔ·ͰʹIngesterϓϩηε͕μ΢ϯ͢Δͱͦͷؒ ͚ͩͦʹ͋ͬͨϩά͸ܽଛ͢Δ ͋Δఔ౓ׂΓ੾ΓΛ͢Δ • WAL͕͋ΔͷͰ࠶ىಈޙʹ͙͢ʹ෮چͰ͖Δ

    • ࡉ͔͍ܽଛΑΓো֐࣌ʹՔಇܧଓͰ͖ΔՄೳੑΛߴΊΔํ ޲ʹৼΔ
  147. 147 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ ରࡦ2. ো֐࣌͸Ұ࣌తʹWALΛແޮʹͯ͠ىಈ͢Δ • WALϩʔυ͕૸Βͳ͍ͷͰɺMemory͔Β͋;ΕΔྔཷ·͍ͬͯ ͯ΋ϓϩηεىಈͰ͖ΔΑ͏ʹͳΔ • ࠶༗ޮʹ͢Δͱ͖ʹϩά͕شൃ͠ͳ͍Α͏ɺreplication factorɺ

    update strategyʹ഑ྀ͢Δ
  148. 148 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ

  149. 149 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Amazon

    S3 Chunk Cache
  150. 150 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ Storageো֐࣌ʹ΋ϩάΛݕࡧ͢ΔͨΊͷ৚݅ • Ingester͕࠷௿Ұͭ͸݈ࡏͰ͋Δ͜ͱ • ݕࡧ݁ՌΛCache͔IngesterͷσʔλͰΧόʔͰ͖Δ͜ͱ • Cacheʹͳ͍࣌ؒൣғΛΫΤϦʹࢦఆ͠ͳ͍͜ͱ

  151. 151 ·ͱΊ

  152. 152 ·ͱΊ LokiͷίΞͰ͋Δॻ͖ࠐΈͱಡΈࠐΈϓϩηεΛৄղ • ಈ࡞ݪཧ͕Θ͔ͬͨ͜ͱͰɺτϥϒϧγϡʔςΟϯά΍νϡ ʔχϯά͕Մೳʹ • Ͳ͜ͰͲͷΤϯίʔσΟϯάͰσʔλΛ͔࣋ͭΛ೺Ѳ͢Δ͜ ͱͰΩϟύγςΟϓϥϯχϯά͕Մೳʹ •

    ো֐࣌ͷڍಈΛ೺Ѳ͢Δ͜ͱͰద੾ͳ४උ͕ݕ౼Մೳʹ
  153. 153 ·ͱΊ ఻͑ΒΕͳ͔ͬͨ͜ͱ • ϩά͔ΒͷϝτϦΫεੜ੒΍ΞϥʔςΟϯά • ϩάͷϦςϯγϣϯ؅ཧʹ͍ͭͯ • LokiࣗମͷϞχλϦϯάʹ͍ͭͯ •

    ֤ίϯϙʔωϯτͷΩϟύγςΟ؅ཧʹ͍ͭͯ • Out of order entry໰୊΁ͷରࡦʹ͍ͭͯ
  154. 154 ผ్ຊΛॻ͖·͢

  155. 155 Twitter: @taisho6339 ࣭͝໰౳͸

  156. 156 THANK YOU