Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

CloudNative Days Tokyo 2021での登壇資料です
https://event.cloudnativedays.jp/cndt2021/talks/1252

LINE Developers

November 04, 2021
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. 2 ࣗݾ঺հ - Title: Senior Software Engineer@LINE Corp - Role:

    Private Cloud։ൃ૊৫ͷSRE - Mission Private CloudΛԣஅͨ͠৴པվળ - Interest: Kubernetes, ෼ࢄγεςϜ, ےτϨ, OSS׆ಈ - Twitter: @taisho6339
  2. like Prometheus but for logs • ϩάͷอଘͱݕࡧػೳ • ϩάϕʔεͷΞϥʔςΟϯάػೳ •

    ϩάϕʔεͷϝτϦΫε࡞੒ػೳ • ϚϧνςφϯτͷDefault Support 3 Lokiͱ͸Կ͔ʁ
  3. 7 Private Cloud “Verda” in LINE is based on OpenStack.

    since 2016~ FaaS PaaS IaaS NAT LB Bare metal
  4. 18 ॻ͖ࠐΈϓϩηε Overview Amazon S3 Chunk Cache Index Write Cache

    Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) • Fluentd΍PromtailͳͲͷɺϩάૹ৴Client • ϩάΛಡΈऔΓLokiͷΤϯυϙΠϯτ΁ૹ৴͢Δ
  5. 19 Amazon S3 Chunk Cache Index Write Cache Loki Clients

    Storage Distributor Ingesters Clients (Promtail, Fluentd) • ϦΫΤετͷόϦσʔγϣϯΛߦ͏ • ద੾ͳIngester΁ϧʔςΟϯά͢Δ ॻ͖ࠐΈϓϩηε Overview
  6. 20 Amazon S3 Chunk Cache Index Write Cache Loki Clients

    Storage Distributor Ingesters Clients (Promtail, Fluentd) ϩάΛ࣮ࡍʹετϨʔδʹอଘ͢Δ Ұఆ࣌ؒόοϑΝϦϯά͔ͯ͠Βετ Ϩʔδʹอଘ ॻ͖ࠐΈϓϩηε Overview
  7. 21 Amazon S3 Loki Clients Storage Distributor Ingesters Clients (Promtail,

    Fluentd) • S3ʹϩάΛӬଓԽ • MemcachedʹϩάͷΩϟογϡΛอଘ Chunk Cache ॻ͖ࠐΈϓϩηε Overview
  8. 23 Client -> Distributor Clients (Promtail, Fluentd) Distributor HTTP Headers

    X-Scope-OrgID : <Tenant ID> TenantIDΛRequest Headerʹهࡌ
  9. 24 Client -> Distributor Clients (Promtail, Fluentd) Distributor {service=“keystone”, hostname=“host1”}

    00:00:02 keystone log body {service=“keystone”, hostname=“host1”} 00:00:03 keystone log body {service=“keystone”, hostname=“host1”} 00:00:04 keystone log body Loki΁ૹΔϩάσʔλߏ଄
  10. 26 Client -> Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body

    ϩά͸͍͔ͭ͘ͷϥϕϧΛ࣋ͭɻ TenantIDͱϥϕϧͷ૊Έ߹Θͤͷ ҰͭҰͭΛɺʮStreamʯͱݺͿ Stream Log Body TS
  11. 32 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor Ingesterಉ࢜ͰΫϥελϦϯά

    • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ • Consistent HashͷRingʹͳͬͯ ͍Δ
  12. 34 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor a6965cd7

    Consistent Hashʹج͍ͮͯɺ ࢉग़ͨ͠Hash஋ʹରԠ͢ΔIngester Λreplication factor෼ཁٻ
  13. 43 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL

    Segment Ingester ΋͠Stream಺Ͱ࠷ޙʹड͚औͬͨϩάͷ࣌ؒΑΓ લͷ࣌ؒͷϩά͕དྷͨ৔߹͸ϦΫΤετΛࣦഊͤ͞Δ Out of order entry error
  14. 46 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk {service=“keystone”, hostname=“host1”}
  15. 47 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk Log Append ड͚औͬͨϩά͸Ұ୴Head΁ Append͞ΕΔ
  16. 48 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢
  17. 49 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block MemoryChunk Log Log Log Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢
  18. 50 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Blockʹ௥Ճ ҰఆαΠζཷ·ͬͨΒ ઃఆͨ͠ܗࣜͰѹॖ͠ɺ blocksʹ௥Ճ͢Δ
  19. 51 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block

    compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Block͕Ұఆ਺ཷ·ͬͨΒ Read Only ModeʹͳΓɺFlush Queue΁ (Default 10 blocks, target chunk sizeͰࢦఆ)
  20. 53 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Chunk Cache Disk సஔIndex
  21. 54 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε ৚݅Λຬͨ͢ChunkΛݕ஌ • ࢦఆαΠζʹ౸ୡ • ࠷ޙͷߋ৽͔Βchunk idle periodܦա • max_chunk_ageܦա
  22. 55 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Disk సஔΠϯσοΫε Flush Queue΁Enqueue
  23. 56 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Disk Enqueue͞Εͨ΋ͷΛFlush 1. ChunkΛS3΁อଘ 2. Chunk Cache΁อଘ(Write Through) 3. సஔIndexΛϩʔΧϧBoltDBʹอଘ Chunk Cache సஔIndex
  24. 57 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Goroutine Goroutine Amazon S3 Disk Ingester΁ͷϦΫΤετ͸ ෳ੡͞Ε͍ͯΔͷͰɺ طʹCacheʹೖ͍ͬͯΔChunk͸ Storage΁ͷॻ͖ࠐΈ͕ൃੜ͠ͳ ͍Α͏ʹ੍ޚ͍ͯ͠Δ Chunk Chunk Cache సஔIndex
  25. 58 Cacheͷෛՙ෼ࢄ Chunk Cache Ingesters Chunk Cache Chunk Cache Chunk1

    Key Chunk2 Key Chunk3 Key ઃఆʹΑͬͯConsistent HashʹΑΓɺ ChunkͷKeyΛݩʹ෼ࢄͯ͠อଘͰ͖Δ
  26. 60 ChunkͷFlush(࠶ܝ) Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε
  27. 61 IngesterͷChunkͷ࣋ͪํ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Flush͞ΕΔલʹϓϩηε͕ࢭ·Δͱ Chunk͕شൃ͢Δ
  28. 65 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Append SegmentϑΝΠϧʹ rawσʔλͷ··௥ه͞ΕΔ
  29. 66 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Create! 1ϑΝΠϧ͕େ͖͘ͳͬͯ͘Δͱ ผͷSegmentϑΝΠϧΛ࡞੒ Segment2
  30. 67 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Goroutine Disk Segment1 Segment2 Ұఆظؒ͝ͱʹ ෆཁͳSegmentϑΝΠϧͷ ύʔδॲཧ͕ೖΔ
  31. 68 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Segment2 Segment3 Create! SegmentΛҰͭਐΊΔ
  32. 69 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 IngesterͷະFlush ChunkΛ Checkpointͱݺ͹ΕΔ εφοϓγϣοτͱͯ͠อଘ
  33. 70 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 MemoryChunkߏ଄ͷ··อଘ͢Δ ͷͰblock͸ѹॖܗࣜɺ headঢ়ଶͷϩά͸ඇѹॖͱͳΔ
  34. 71 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment3 Checkpoint1 ࠷৽ͷCheckpointͱSegmentΛ࢒ ͯ͢͠΂ͯ࡟আ͢Δ
  35. 72 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Goroutine Disk Segment3 Checkpoint1 ͦͷ࣌఺ͰFlush͞Ε͍ͯͳ͍ ͢΂ͯͷϩάؚ͕·Ε͍ͯΔ
  36. 73 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ
  37. 75 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Stream1 Stream2 Stream1 Stream2 ෮ݩ͕׬ྃ͢Δ·Ͱ͸ϓϩηεΛىಈ͠ͳ͍
  38. 76 Ingester্ͷσʔλͱEncodeܗࣜ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Disk Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷
  39. 79 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend Queriers Query Result Cache Ingesters
  40. 81 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ड͚औͬͨΫΤϦΛ࣌ؒͳͲͰ ෼ׂͯ͠ΩϡʔΠϯά͢Δ
  41. 82 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ෳ਺ͷQuerier͕Ωϡʔ͔ΒQuery Λड͚औΓϋϯυϦϯά͢Δ Queriers
  42. 83 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ͢΂ͯͷIngesterʹରͯ͠ɺ MemoryChunk͔ΒQueryʹMatch ͢Δ΋ͷΛཁٻ Queriers Ingesters
  43. 84 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend QueryʹରԠ͢ΔసஔIndexΛऔಘ͢Δ ͜ͷͱ͖CacheʹಁաతʹΞΫηε Queriers
  44. 85 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend Cacheʹଘࡏ͠ͳ͍৔߹͸ɺ ϩʔΧϧͷBoltDB͔ΒMatch͢ΔIndexΛऔಘ ͦͯ݁͠ՌΛCacheʹอଘ͢Δ(snappy) Queriers BoltDB
  45. 86 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend సஔIndex͔Βର৅ͷChunkΛׂΓग़͢ Queriers
  46. 87 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ChunkΛऔಘ͢Δ Cacheʹͳ͍΋ͷ͸Storage͔Βऔಘ͠ɺ Cacheʹอଘ͢Δ Queriers
  47. 88 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ͢΂ͯͷQuerier͔Βͷ݁ՌΛड͚औΓɺ ू໿ɺιʔτɺॏෳഉআΛ࣮ࢪ Queriers
  48. 89 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend ݁ՌΛQuery Result Cache΁อଘ͢Δ Queriers Query Result Cache
  49. 90 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki

    Storage Query Frontend Ϩεϙϯεฦ٫ Queriers Query Result Cache
  50. 94 SeriesID {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body key, valueͷ૊Έ߹Θͤͷsha256

    9ac2adda49e899b312a9abb895656b1ab26c9858fd500f2ae3983d5309b39363/ ϩά SeriesID
  51. Hash Value Range Value Value TenantID + LabelName Hash(Label Value)

    + SeriesID Label Value సஔIndexͷߏ଄Πϝʔδ Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index
  52. Hash Value Range Value Value TenantID + LabelName Hash(Label Value)

    + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index Hash + RangeͰϢχʔΫߦΛಛఆ͢Δ సஔIndexͷߏ଄Πϝʔδ
  53. Hash Value Range Value Value TenantID + LabelName Hash(Label Value)

    + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index సஔIndexͷߏ଄Πϝʔδ Range Value͸ൣғݕࡧɺιʔτʹར༻Ͱ͖Δ
  54. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value)

    + SeriesID) Value (Label Value) Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler ςʔϒϧΠϝʔδ సஔIndexͷߏ଄Πϝʔδ
  55. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value)

    + SeriesID) Value Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”} ώοτ͢Δͷ͸͜ͷϨίʔυ
  56. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value)

    + SeriesID) Value Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”} ର৅SeriesID
  57. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min Hash Value (TenantID

    + SeriesID) Range Value (Chunk։࢝࣌ؒ + Chunk Key) Value Tenant1:c79abadeff 1635252768:chunk1 nil Tenant1:c79abadeff 1635256368:chunk2 nil Tenant1:c79abadeff 1635260768:chunk3 nil Index͔ΒͷChunk KeyׂΓग़͠
  58. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min Hash Value (TenantID

    + SeriesID) Range Value (Chunk։࢝࣌ؒ + Chunk Key) Value Tenant1:c79abadeff 1635252768:chunk1 nil Tenant1:c79abadeff 1635256368:chunk2 nil Tenant1:c79abadeff 1635260768:chunk3 nil Index͔ΒͷChunk KeyׂΓग़͠ ର৅ChunkͷRecord
  59. 115 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Ϩεϙϯε͕نఆ݅਺ʹୡ͢Δ·Ͱɺ

    IteratorΛಡΈࠐΜͰ͍͘ ϩάͷFilter৚݅͸͜͜ͰධՁ͞ΕΔ |= “level=ERROR”
  60. 116 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response LazyChunkΛಡΈࠐΉ৔߹͸ɺ

    Ωϟογϡ΁ɺͳ͚Ε͹Storage΁ ChunkΛ໰͍߹ΘͤΔ Amazon S3 Chunk Cache
  61. 119 Queryͷ෼ׂઓུ 1. ࣌ؒ͝ͱʹ෼ׂ͢Δ • 1࣌ؒ෼ͷϩάΛݕࡧ͢Δ৔߹ɺ15෼Ͱ෼ׂ͢ΔઃఆͳΒ4 ͭͷΫΤϦʹ෼ղ͞Ε࣮ͯߦ͞ΕΔ 2. సஔIndexΛSharding͢Δ •

    ͋Β͔͡ΊసஔIndexʹShard൪߸Λ͍Ε͓͖ͯɺ QueryFrontend͕QueryΛ෼ׂ͠ɺͦΕͧΕʹShard൪߸Λ ૠೖͯ͠QuerierʹΘͨ͢
  62. Hash Value Range Value Value TenantID + LabelName Shard Number

    + Hash(Label Value) + SeriesID Label Value సஔIndex΁ͷShard൪߸ຒΊࠐΈ Shard Number = SeriesID % shard count
  63. 123 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”}

    |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys औΕΔChunk͕shardͰ෼ׂ͞ΕΔ
  64. 124 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”}

    |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys |= “level=ERROR”ͷfilterॲཧΛ෼ׂॲཧͰ͖Δ
  65. 127 BoltDB Shipper - Ingester side Ingester Shipper Disk Querier

    Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 Ұఆ࣌ؒ͝ͱʹϩʔΧϧDiskʹ͋ΔIndex ΛS3΁Ξοϓϩʔυ͢Δ Ξοϓϩʔυޙʹ࡟আ͢Δ
  66. 128 BoltDB Shipper - Querier side Ingester Shipper Disk Querier

    Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 • ىಈ࣌ʹS3ʹ͋ΔIndexΛDownload • Query࣌ʹ଍Γͳ͍Index͸S3͔Β౎౓ μ΢ϯϩʔυ • Ұఆ࣌ؒ͝ͱʹ࠷ऴ࢖༻͔Β CacheTTLܦաͨ͠IndexΛ࡟আ
  67. 129 BoltDB Shipper Ingester Shipper Disk Querier Shipper Disk Index

    1 Index 2 Index 1 Index 2 Amazon S3 Ingester, Querier͸ϩʔΧϧͰIndexΛѻ͍ɺ Shipper͕ඇಉظͰIndexΛStorageͱಉظ͢Δ
  68. 131 Name ໾ׂ λΠϓ σʔλͷ࣋ͪԽ ΫϥελϦϯά༗ແ Distributor όϦσʔγϣϯͱIngester΁ͷϧʔςΟϯά Stateless ༗

    Ingester σʔλͷόοϑΝϦϯάͱFlush Stateful Memory: Chunks(raw + ѹॖ) Disk: WAL(raw + ѹॖ) సஔIndex(ѹॖ) ༗ Query Frontend ΫΤϦͷ෼ׂɺΩϡʔ੍ޚ Stateless ແ Querier ΫΤϦͷ࣮ߦ Stateful Disk: సஔIndexͷCache(ѹॖ) ແ Chunk Cache ChunkͷΩϟογϡ Stateful Memory: Chunks(ѹॖ) ༗(ΫϥΠΞϯταΠυ) Index Read Cache IndexͷRead༻Ωϟογϡ Stateful Memory: సஔIndex(Snappy) ༗(ΫϥΠΞϯταΠυ) Index Write Cache ಉ͡Indexͷॻ͖ࠐΈ͕ෳ਺ൃੜ͠ͳ͍Α͏ʹ ͢ΔͨΊͷ੍ޚ༻Ωϟογϡ (BoltDB ShipperͰ͸ෆཁ) Stateful Memory: Chunk Key(raw) ༗(ΫϥΠΞϯταΠυ) Query Result Cache ΫΤϦͷ݁ՌͷΩϟογϡ Stateful Memory: Query Result(raw) ༗(ΫϥΠΞϯταΠυ)
  69. 135 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk Segment1 Checkpoint1 ͕ࣦ͜͜ഊ సஔIndex
  70. 136 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 Flush͸ඇಉظͳͷͰ ॻ͖ࠐΈࣗମ͸ࣦഊ͠ͳ͍
  71. 137 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ MemoryͱDiskʹͨ·Γଓ͚Δ
  72. 138 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ ઌʹMemory͕͋;ΕɺOOM
  73. 139 WALͷ࢓૊Έ(࠶ܝ) Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1

    Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ
  74. 141 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 WAL͸࣮࣭MemoryͷSnapshot ϩʔυ੒ޭͯ͠΋͍ۙ͏ͪʹOOM ϩʔυࣦഊͨ͠Βͦ΋ͦ΋ىಈ͠ͳ͍
  75. 142 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2

    Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ϘτϧωοΫ͸IngesterͷMemory
  76. 143 Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Disk

    Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷ Ingester্ͷσʔλͱEncodeܗࣜ(࠶ܝ)
  77. 145 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ 1೔ͷྲྀྔ10TBͷ؀ڥͰ1࣌ؒ଱͑Δ(ฐࣾͷ1Ϧʔδϣϯ) 1000 / 24 = 41.6 GB /

    hour • Replication Factor = 1 • Ingester * 11୆ • Memory: 4GiB • Disk: 8GiB(ϚʔδϯΛऔͬͯMemoryͷ2ഒ) • Chunk Cache * 14୆ • Memory: 3GiB