Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

CloudNative Days Tokyo 2021での登壇資料です
https://event.cloudnativedays.jp/cndt2021/talks/1252

1. 1 Grafana LokiͰߏங͢Δ େن໛ϩάϞχλϦϯάج൫ CNDT2021 LINEגࣜձࣾ Hiroki Sakamoto / @taisho6339
2. 2 ࣗݾ঺հ - Title: Senior Software Engineer@LINE Corp - Role: Private Cloud։ൃ૊৫ͷSRE - Mission Private CloudΛԣஅͨ͠৴པվળ - Interest: Kubernetes, ෼ࢄγεςϜ, ےτϨ, OSS׆ಈ - Twitter: @taisho6339
3. like Prometheus but for logs • ϩάͷอଘͱݕࡧػೳ • ϩάϕʔεͷΞϥʔςΟϯάػೳ • ϩάϕʔεͷϝτϦΫε࡞੒ػೳ • ϚϧνςφϯτͷDefault Support 3 Lokiͱ͸Կ͔ʁ
4. 4 Lokiͱ͸Կ͔ʁ
5. 5 Lokiͱ͸Կ͔ʁ
6. 6 ҆͘େ༰ྔͷϩάΛอଘՄೳ
7. 7 Private Cloud “Verda” in LINE is based on OpenStack. since 2016~ FaaS PaaS IaaS NAT LB Bare metal
8. 8 Private Cloud “Verda” in LINE Virtual Machines 74000+ 30000+ 4000+ Physical Machines Hypervisors
9. 9 20 TB / day application logs
10. 10 Loki is suitable for us!
11. 11 Lokiͷ೉͠͞ Loki͸ϚΠΫϩαʔϏε • ֤ίϯϙʔωϯτɺ֤Ωϟογϡͷ࢓૊Έͱ໾ׂ͕ෆ໌ྎ • ϩάσʔλ͸Ͳ͜ͰͲΜͳܗࣜͰͲͷ͘Β͍อ࣋͞ΕΔ͔ෆ໌ྎ • ετϨʔδো֐࣌͸Ͳ͏͍͏ڍಈʹͳΔͷ͔ෆ໌ྎ • ຊ൪Ͱӡ༻͢ΔͳΒԿΛߟྀ͠ͳ͍ͱ͍͚ͳ͍ͷ͔͕ෆ໌ྎ
12. 12 ຊηογϣϯͷΰʔϧ ࠃ಺Ͱ࠷΋ৄࡉʹղઆ͢Δ͜ͱΛ໨ࢦ͠·͢ • ମܥతʹLokiͷίϯϙʔωϯτͷ໾ׂͱ࢓૊ΈΛ஌Δ • τϥϒϧ࣌ʹ΋ݪҼͷಛఆ͕ਝ଎ʹͰ͖ΔΑ͏ʹͳΔ • ࣗྗͰΩϟύγςΟ؅ཧɺύϑΥʔϚϯενϡʔχϯάͰ͖ΔΑ ͏ʹͳΔ
13. 13 ຊηογϣϯ૝ఆߏ੒ Loki Version: v2.3.0 Cache: Memcached Chunk Storage: AWS S3 Index Storage: AWS S3 + BoltDB Shipper
14. 14 1) ϩάͷॻ͖ࠐΈϓϩηεΛ஌Δ 2) ϩάͷಡΈࠐΈϓϩηεΛ஌Δ 3) ো֐࣌ͷڍಈΛ஌Δ ຊηογϣϯͷRoadmap
15. 15 1) ϩάͷॻ͖ࠐΈϓϩηε
16. 16 Overview
17. 17 ॻ͖ࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd)
18. 18 ॻ͖ࠐΈϓϩηε Overview Amazon S3 Chunk Cache Index Write Cache Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) • Fluentd΍PromtailͳͲͷɺϩάૹ৴Client • ϩάΛಡΈऔΓLokiͷΤϯυϙΠϯτ΁ૹ৴͢Δ
19. 19 Amazon S3 Chunk Cache Index Write Cache Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) • ϦΫΤετͷόϦσʔγϣϯΛߦ͏ • ద੾ͳIngester΁ϧʔςΟϯά͢Δ ॻ͖ࠐΈϓϩηε Overview
20. 20 Amazon S3 Chunk Cache Index Write Cache Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) ϩάΛ࣮ࡍʹετϨʔδʹอଘ͢Δ Ұఆ࣌ؒόοϑΝϦϯά͔ͯ͠Βετ Ϩʔδʹอଘ ॻ͖ࠐΈϓϩηε Overview
21. 21 Amazon S3 Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) • S3ʹϩάΛӬଓԽ • MemcachedʹϩάͷΩϟογϡΛอଘ Chunk Cache ॻ͖ࠐΈϓϩηε Overview
22. 22 Client͔ΒDistributor΁ͷૹ৴
23. 23 Client -> Distributor Clients (Promtail, Fluentd) Distributor HTTP Headers X-Scope-OrgID : <Tenant ID> TenantIDΛRequest Headerʹهࡌ
24. 24 Client -> Distributor Clients (Promtail, Fluentd) Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body {service=“keystone”, hostname=“host1”} 00:00:03 keystone log body {service=“keystone”, hostname=“host1”} 00:00:04 keystone log body Loki΁ૹΔϩάσʔλߏ଄
25. 25 Client -> Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body Stream Log Body TS
26. 26 Client -> Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body ϩά͸͍͔ͭ͘ͷϥϕϧΛ࣋ͭɻ TenantIDͱϥϕϧͷ૊Έ߹Θͤͷ ҰͭҰͭΛɺʮStreamʯͱݺͿ Stream Log Body TS
27. 27 DistributorͰͷόϦσʔγϣϯ • ϥϕϧͷܗࣜ͸ਖ਼͍͔͠ʁ • Rate limitΛ௒͑ͳ͍͔ʁ Clients (Promtail, Fluentd) Distributor
28. 28 Distributor DistributorͰͷόϦσʔγϣϯ Distributor Distributor Distributor Distributorಉ࢜ͰΫϥελϦϯά • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ • ingestion_rate_strategy͕global ͳΒΫϥελશମͰingestion rateΛ੍ޚ͢Δ
29. 29 Distributor DistributorͰͷόϦσʔγϣϯ Distributor Distributor Distributor DistributorશମͰόϦσʔγϣϯ ͢ΔͨΊͷΫϥελϦϯά
30. 30 Distributor͔ΒIngester΁ͷૹ৴
31. 31 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ݕࡧͷͨΊʹ(ޙड़) ෳ਺ͷIngesterʹ ৑௕Խͯ͠ϩάΛૹΔ
32. 32 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor Ingesterಉ࢜ͰΫϥελϦϯά • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ • Consistent HashͷRingʹͳͬͯ ͍Δ
33. 33 Distributor -> Ingesters Ingester Ingesters Ingester Ingester FNV1-32bitͰHash஋Λੜ੒ tenantID + {service=“keystone”, hostname=“host1”} a6965cd7
34. 34 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor a6965cd7 Consistent Hashʹج͍ͮͯɺ ࢉग़ͨ͠Hash஋ʹରԠ͢ΔIngester Λreplication factor෼ཁٻ
35. 35 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ؼ͖ͬͯͨෳ਺ͷIngesterʹɺ ಉ͡ϩάΛಉ࣌ૹ৴
36. 36 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor OK OK Fail
37. 37 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ա൒਺͕OKͳΒ੒ޭ OK OK Fail
38. 38 IngesterͷRequest Handling
39. 39 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 {service=“keystone”, hostname=“host1”} Ingester
40. 40 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks StreamͰGrouping͞Εɺ Chunkͱ͍͏ܗࣜͰAppend Ingester
41. 41 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL Segment Ingester WALʹه࿥
42. 42 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL Segment Ingester OK
43. 43 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL Segment Ingester ΋͠Stream಺Ͱ࠷ޙʹड͚औͬͨϩάͷ࣌ؒΑΓ લͷ࣌ؒͷϩά͕དྷͨ৔߹͸ϦΫΤετΛࣦഊͤ͞Δ Out of order entry error
44. 44 ChunkόοϑΝϦϯά
45. 45 IngesterͷChunkόοϑΝϦϯά • Ұఆ਺ͷϩάΛStream͝ͱʹ ʮChunkʯʹ·ͱΊΔ • Chunk͸ϝϞϦ্ʹอଘ͞ΕΔ • HeadɺBlocksͱ͍͏഑ྻΛอ࣋ Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk
46. 46 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk {service=“keystone”, hostname=“host1”}
47. 47 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk Log Append ड͚औͬͨϩά͸Ұ୴Head΁ Append͞ΕΔ
48. 48 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢
49. 49 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk Log Log Log Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢
50. 50 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Blockʹ௥Ճ ҰఆαΠζཷ·ͬͨΒ ઃఆͨ͠ܗࣜͰѹॖ͠ɺ blocksʹ௥Ճ͢Δ
51. 51 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Block͕Ұఆ਺ཷ·ͬͨΒ Read Only ModeʹͳΓɺFlush Queue΁ (Default 10 blocks, target chunk sizeͰࢦఆ)
52. 52 Ingester͔ΒStorage΁Flush
53. 53 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Chunk Cache Disk సஔIndex
54. 54 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε ৚݅Λຬͨ͢ChunkΛݕ஌ • ࢦఆαΠζʹ౸ୡ • ࠷ޙͷߋ৽͔Βchunk idle periodܦա • max_chunk_ageܦա
55. 55 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Disk సஔΠϯσοΫε Flush Queue΁Enqueue
56. 56 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Disk Enqueue͞Εͨ΋ͷΛFlush 1. ChunkΛS3΁อଘ 2. Chunk Cache΁อଘ(Write Through) 3. సஔIndexΛϩʔΧϧBoltDBʹอଘ Chunk Cache సஔIndex
57. 57 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Goroutine Goroutine Amazon S3 Disk Ingester΁ͷϦΫΤετ͸ ෳ੡͞Ε͍ͯΔͷͰɺ طʹCacheʹೖ͍ͬͯΔChunk͸ Storage΁ͷॻ͖ࠐΈ͕ൃੜ͠ͳ ͍Α͏ʹ੍ޚ͍ͯ͠Δ Chunk Chunk Cache సஔIndex
58. 58 Cacheͷෛՙ෼ࢄ Chunk Cache Ingesters Chunk Cache Chunk Cache Chunk1 Key Chunk2 Key Chunk3 Key ઃఆʹΑͬͯConsistent HashʹΑΓɺ ChunkͷKeyΛݩʹ෼ࢄͯ͠อଘͰ͖Δ
59. 59 Write Ahead Log
60. 60 ChunkͷFlush(࠶ܝ) Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε
61. 61 IngesterͷChunkͷ࣋ͪํ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Flush͞ΕΔલʹϓϩηε͕ࢭ·Δͱ Chunk͕شൃ͢Δ
62. 62 WALͷҙٛ ӬଓԽͯ͠Memoryͷشൃʹඋ͑Δ
63. 63 LokiͷWALͷಛ௃ • ϩάड৴࣌ʹɺMemoryͱWAL྆ํʹॻ͖ࠐΉ • WALॻ͖ࠐΈ͕ࣦഊͯ͠΋ॲཧΛࣦഊͤ͞ͳ͍ • Ingesterͷϓϩηεىಈ࣌ʹWAL͔Βͷ෮چॲཧ͕ೖΔ • Ұఆظؒ͝ͱʹෆཁͳWAL͸ύʔδ͞ΕΔ
64. 64 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Log Entry ϩάΛૹ৴
65. 65 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Append SegmentϑΝΠϧʹ rawσʔλͷ··௥ه͞ΕΔ
66. 66 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Create! 1ϑΝΠϧ͕େ͖͘ͳͬͯ͘Δͱ ผͷSegmentϑΝΠϧΛ࡞੒ Segment2
67. 67 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Goroutine Disk Segment1 Segment2 Ұఆظؒ͝ͱʹ ෆཁͳSegmentϑΝΠϧͷ ύʔδॲཧ͕ೖΔ
68. 68 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Segment2 Segment3 Create! SegmentΛҰͭਐΊΔ
69. 69 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 IngesterͷະFlush ChunkΛ Checkpointͱݺ͹ΕΔ εφοϓγϣοτͱͯ͠อଘ
70. 70 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 MemoryChunkߏ଄ͷ··อଘ͢Δ ͷͰblock͸ѹॖܗࣜɺ headঢ়ଶͷϩά͸ඇѹॖͱͳΔ
71. 71 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment3 Checkpoint1 ࠷৽ͷCheckpointͱSegmentΛ࢒ ͯ͢͠΂ͯ࡟আ͢Δ
72. 72 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment3 Checkpoint1 ͦͷ࣌఺ͰFlush͞Ε͍ͯͳ͍ ͢΂ͯͷϩάؚ͕·Ε͍ͯΔ
73. 73 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ
74. 74 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Stream1 Stream2 Stream1 Stream2 Memory಺ʹ෮ݩ͢Δ
75. 75 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Stream1 Stream2 Stream1 Stream2 ෮ݩ͕׬ྃ͢Δ·Ͱ͸ϓϩηεΛىಈ͠ͳ͍
76. 76 Ingester্ͷσʔλͱEncodeܗࣜ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Disk Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷
77. 77 2) ϩάͷಡΈࠐΈϓϩηε
78. 78 Overview
79. 79 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend Queriers Query Result Cache Ingesters
80. 80 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ϦΫΤετΛड෇͚Δ
81. 81 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ड͚औͬͨΫΤϦΛ࣌ؒͳͲͰ ෼ׂͯ͠ΩϡʔΠϯά͢Δ
82. 82 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ෳ਺ͷQuerier͕Ωϡʔ͔ΒQuery Λड͚औΓϋϯυϦϯά͢Δ Queriers
83. 83 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ͢΂ͯͷIngesterʹରͯ͠ɺ MemoryChunk͔ΒQueryʹMatch ͢Δ΋ͷΛཁٻ Queriers Ingesters
84. 84 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend QueryʹରԠ͢ΔసஔIndexΛऔಘ͢Δ ͜ͷͱ͖CacheʹಁաతʹΞΫηε Queriers
85. 85 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend Cacheʹଘࡏ͠ͳ͍৔߹͸ɺ ϩʔΧϧͷBoltDB͔ΒMatch͢ΔIndexΛऔಘ ͦͯ݁͠ՌΛCacheʹอଘ͢Δ(snappy) Queriers BoltDB
86. 86 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend సஔIndex͔Βର৅ͷChunkΛׂΓग़͢ Queriers
87. 87 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ChunkΛऔಘ͢Δ Cacheʹͳ͍΋ͷ͸Storage͔Βऔಘ͠ɺ Cacheʹอଘ͢Δ Queriers
88. 88 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ͢΂ͯͷQuerier͔Βͷ݁ՌΛड͚औΓɺ ू໿ɺιʔτɺॏෳഉআΛ࣮ࢪ Queriers
89. 89 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ݁ՌΛQuery Result Cache΁อଘ͢Δ Queriers Query Result Cache
90. 90 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend Ϩεϙϯεฦ٫ Queriers Query Result Cache
91. 91 సஔIndex͔Βର৅Chunkͷબఆ
92. 92 సஔIndexͷ໨త S3͔ΒChunkΛ࠷খ࿑ྗͰऔಘ͢Δ͜ͱ
93. 93 సஔIndex͔Βର৅ChunkͷߜΓࠐΈ 1.LabelͷKeyͱValueͷ૊Έ߹Θ͔ͤΒStreamͷID (Series ID)Λऔಘ͢Δ 2.Series IDͱ࣌ؒൣғ͔ΒChunkͷKeyΛऔಘ͢Δ 3.ChunkͷKey͔ΒS3্ɺMemcached্ͷύεΛׂ Γग़͠ɺ௚઀ChunkΛDownload
94. 94 SeriesID {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body key, valueͷ૊Έ߹Θͤͷsha256 9ac2adda49e899b312a9abb895656b1ab26c9858fd500f2ae3983d5309b39363/ ϩά SeriesID
95. 95 Chunk Key {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body Tenant1/a6965cd7:Chunk։࢝࣌ؒ:Chunkऴྃ࣌ؒ key, valueͷ૊Έ߹ΘͤͷHash஋ a6965cd7 ϩά FingerPrint Chunk Key
96. Hash Value Range Value Value TenantID + LabelName Hash(Label Value) + SeriesID Label Value సஔIndexͷߏ଄Πϝʔδ Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index
97. Hash Value Range Value Value TenantID + LabelName Hash(Label Value) + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index Hash + RangeͰϢχʔΫߦΛಛఆ͢Δ సஔIndexͷߏ଄Πϝʔδ
98. Hash Value Range Value Value TenantID + LabelName Hash(Label Value) + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index సஔIndexͷߏ଄Πϝʔδ Range Value͸ൣғݕࡧɺιʔτʹར༻Ͱ͖Δ
99. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value) + SeriesID) Value (Label Value) Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler ςʔϒϧΠϝʔδ సஔIndexͷߏ଄Πϝʔδ
100. ͜ͷIndex͸ LabelͷKeyͱValueͷύλʔϯ෼͚ͩ࡞ΒΕΔ ΧʔσΟφϦςΟͷߴ͍ϥϕϧ͸ ͜ͷIndex͕େྔʹ࡞ΒΕΔ͜ͱʹͳΔ సஔIndexͷߏ଄Πϝʔδ
101. SeriesID͔ΒChunk KeyΛҾ͘Index సஔIndexͷߏ଄Πϝʔδ Hash Value Range Value Value TenantID + SeriesID Chunkͷ։࢝࣌ؒ + Chunk Key nil
102. 102 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} |= “level=ERROR” LogQL ϥϕϧϚον෦ ϑΟϧλ෦
103. 103 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} |= “level=ERROR” LogQL ϥϕϧϚον෦ ϑΟϧλ෦ IndexΛ࢖͏ͷ͸͜ͷ෦෼ͷධՁ
104. 104 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} ෼ׂ
105. 105 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} ͦΕͧΕͰϚονϯά৚݅ ͔ΒసஔIndexΛऔಘ సஔIndex సஔIndex
106. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value) + SeriesID) Value Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”} ώοτ͢Δͷ͸͜ͷϨίʔυ
107. Hash Value (TenantID + Label Name) Range Value (Hash(Label Value) + SeriesID) Value Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”} ର৅SeriesID
108. 108 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} Index͔ΒSeriesIDΛநग़ ྆ํʹڞ௨͢Δ΋ͷ͚ͩ࠾༻ SeriesIDs సஔIndex సஔIndex
109. 109 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} SeriesIDͱ࣌ؒൣғ͔Β Chunk KeyΛநग़ SeriesIDs సஔIndex Chunk Keys సஔIndex సஔIndex
110. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min Hash Value (TenantID + SeriesID) Range Value (Chunk։࢝࣌ؒ + Chunk Key) Value Tenant1:c79abadeff 1635252768:chunk1 nil Tenant1:c79abadeff 1635256368:chunk2 nil Tenant1:c79abadeff 1635260768:chunk3 nil Index͔ΒͷChunk KeyׂΓग़͠
111. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min Hash Value (TenantID + SeriesID) Range Value (Chunk։࢝࣌ؒ + Chunk Key) Value Tenant1:c79abadeff 1635252768:chunk1 nil Tenant1:c79abadeff 1635256368:chunk2 nil Tenant1:c79abadeff 1635260768:chunk3 nil Index͔ΒͷChunk KeyׂΓग़͠ ର৅ChunkͷRecord
112. 112 Ϩεϙϯεͷੜ੒ͱฦ٫
113. 113 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks ॳظঢ়ଶͰ͸࣮ମΛ࣋ͨͣɺ ಡΈࠐΈ໋ྩ͕͞ΕͨλΠϛϯάͰετϨʔδ(Ωϟογϡ) ʹChunkΛऔΓʹߦ͘Lazy ChunkΛੜ੒
114. 114 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Ingester͔ΒͷChunkΛ͋Θͤͯɺ IteratorΛੜ੒
115. 115 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Ϩεϙϯε͕نఆ݅਺ʹୡ͢Δ·Ͱɺ IteratorΛಡΈࠐΜͰ͍͘ ϩάͷFilter৚݅͸͜͜ͰධՁ͞ΕΔ |= “level=ERROR”
116. 116 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response LazyChunkΛಡΈࠐΉ৔߹͸ɺ Ωϟογϡ΁ɺͳ͚Ε͹Storage΁ ChunkΛ໰͍߹ΘͤΔ Amazon S3 Chunk Cache
117. 117 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Storage͔ΒಡΈࠐΜͩ৔߹͸ɺ औಘޙʹCacheʹอଘ͢Δ Amazon S3 Chunk Cache
118. 118 Query Sharding
119. 119 Queryͷ෼ׂઓུ 1. ࣌ؒ͝ͱʹ෼ׂ͢Δ • 1࣌ؒ෼ͷϩάΛݕࡧ͢Δ৔߹ɺ15෼Ͱ෼ׂ͢ΔઃఆͳΒ4 ͭͷΫΤϦʹ෼ղ͞Ε࣮ͯߦ͞ΕΔ 2. సஔIndexΛSharding͢Δ • ͋Β͔͡ΊసஔIndexʹShard൪߸Λ͍Ε͓͖ͯɺ QueryFrontend͕QueryΛ෼ׂ͠ɺͦΕͧΕʹShard൪߸Λ ૠೖͯ͠QuerierʹΘͨ͢
120. Hash Value Range Value Value TenantID + LabelName Shard Number + Hash(Label Value) + SeriesID Label Value సஔIndex΁ͷShard൪߸ຒΊࠐΈ Shard Number = SeriesID % shard count
121. 121 సஔIndex΁ͷShard൪߸ຒΊࠐΈ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 Shard Number Stream
122. 122 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”} |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 QueryΛShardͰ෼ׂ
123. 123 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”} |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys औΕΔChunk͕shardͰ෼ׂ͞ΕΔ
124. 124 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”} |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys |= “level=ERROR”ͷfilterॲཧΛ෼ׂॲཧͰ͖Δ
125. 125 BoltDB ShipperʹΑΔIndex؅ཧ
126. 126 BoltDB Shipper Ingester Shipper Disk Querier Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3
127. 127 BoltDB Shipper - Ingester side Ingester Shipper Disk Querier Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 Ұఆ࣌ؒ͝ͱʹϩʔΧϧDiskʹ͋ΔIndex ΛS3΁Ξοϓϩʔυ͢Δ Ξοϓϩʔυޙʹ࡟আ͢Δ
128. 128 BoltDB Shipper - Querier side Ingester Shipper Disk Querier Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 • ىಈ࣌ʹS3ʹ͋ΔIndexΛDownload • Query࣌ʹ଍Γͳ͍Index͸S3͔Β౎౓ μ΢ϯϩʔυ • Ұఆ࣌ؒ͝ͱʹ࠷ऴ࢖༻͔Β CacheTTLܦաͨ͠IndexΛ࡟আ
129. 129 BoltDB Shipper Ingester Shipper Disk Querier Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 Ingester, Querier͸ϩʔΧϧͰIndexΛѻ͍ɺ Shipper͕ඇಉظͰIndexΛStorageͱಉظ͢Δ
130. 130 ֤ίϯϙʔωϯτͷ໾ׂ·ͱΊ
131. 131 Name ໾ׂ λΠϓ σʔλͷ࣋ͪԽ ΫϥελϦϯά༗ແ Distributor όϦσʔγϣϯͱIngester΁ͷϧʔςΟϯά Stateless ༗ Ingester σʔλͷόοϑΝϦϯάͱFlush Stateful Memory: Chunks(raw + ѹॖ) Disk: WAL(raw + ѹॖ) సஔIndex(ѹॖ) ༗ Query Frontend ΫΤϦͷ෼ׂɺΩϡʔ੍ޚ Stateless ແ Querier ΫΤϦͷ࣮ߦ Stateful Disk: సஔIndexͷCache(ѹॖ) ແ Chunk Cache ChunkͷΩϟογϡ Stateful Memory: Chunks(ѹॖ) ༗(ΫϥΠΞϯταΠυ) Index Read Cache IndexͷRead༻Ωϟογϡ Stateful Memory: సஔIndex(Snappy) ༗(ΫϥΠΞϯταΠυ) Index Write Cache ಉ͡Indexͷॻ͖ࠐΈ͕ෳ਺ൃੜ͠ͳ͍Α͏ʹ ͢ΔͨΊͷ੍ޚ༻Ωϟογϡ (BoltDB ShipperͰ͸ෆཁ) Stateful Memory: Chunk Key(raw) ༗(ΫϥΠΞϯταΠυ) Query Result Cache ΫΤϦͷ݁ՌͷΩϟογϡ Stateful Memory: Query Result(raw) ༗(ΫϥΠΞϯταΠυ)
132. 132 3) ো֐࣌ͷڍಈΛ஌Δ
133. 133 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
134. 134 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ S3ো֐࣌ʹඋ͑Δ
135. 135 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk Segment1 Checkpoint1 ͕ࣦ͜͜ഊ సஔIndex
136. 136 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 Flush͸ඇಉظͳͷͰ ॻ͖ࠐΈࣗମ͸ࣦഊ͠ͳ͍
137. 137 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ MemoryͱDiskʹͨ·Γଓ͚Δ
138. 138 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ ઌʹMemory͕͋;ΕɺOOM
139. 139 WALͷ࢓૊Έ(࠶ܝ) Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ
140. 140 WALͷ࢓૊Έ(࠶ܝ) Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Stream1 Stream2 Stream1 Stream2 Memory಺ʹ෮ݩ͢Δ
141. 141 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 WAL͸࣮࣭MemoryͷSnapshot ϩʔυ੒ޭͯ͠΋͍ۙ͏ͪʹOOM ϩʔυࣦഊͨ͠Βͦ΋ͦ΋ىಈ͠ͳ͍
142. 142 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ϘτϧωοΫ͸IngesterͷMemory
143. 143 Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Disk Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷ Ingester্ͷσʔλͱEncodeܗࣜ(࠶ܝ)
144. 144 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ ରࡦ1. ଱͍͑ͨ࣌ؒ෼ͷMemoryΛੵΉ • ࣌ؒ͋ͨΓͷϩάྔ / ѹॖൺ * ࣌ؒ * replication factor ※ѹॖൺ͸ฐ؀ڥͰ͸gzipѹॖͰ10~18ഒͷѹॖൺ
145. 145 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ 1೔ͷྲྀྔ10TBͷ؀ڥͰ1࣌ؒ଱͑Δ(ฐࣾͷ1Ϧʔδϣϯ) 1000 / 24 = 41.6 GB / hour • Replication Factor = 1 • Ingester * 11୆ • Memory: 4GiB • Disk: 8GiB(ϚʔδϯΛऔͬͯMemoryͷ2ഒ) • Chunk Cache * 14୆ • Memory: 3GiB
146. 146 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Replication Factor͸1Ͱ͍͍ͷ͔ʁ • Flush͞ΕΔ·ͰʹIngesterϓϩηε͕μ΢ϯ͢Δͱͦͷؒ ͚ͩͦʹ͋ͬͨϩά͸ܽଛ͢Δ ͋Δఔ౓ׂΓ੾ΓΛ͢Δ • WAL͕͋ΔͷͰ࠶ىಈޙʹ͙͢ʹ෮چͰ͖Δ • ࡉ͔͍ܽଛΑΓো֐࣌ʹՔಇܧଓͰ͖ΔՄೳੑΛߴΊΔํ ޲ʹৼΔ
147. 147 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ ରࡦ2. ো֐࣌͸Ұ࣌తʹWALΛແޮʹͯ͠ىಈ͢Δ • WALϩʔυ͕૸Βͳ͍ͷͰɺMemory͔Β͋;ΕΔྔཷ·͍ͬͯ ͯ΋ϓϩηεىಈͰ͖ΔΑ͏ʹͳΔ • ࠶༗ޮʹ͢Δͱ͖ʹϩά͕شൃ͠ͳ͍Α͏ɺreplication factorɺ update strategyʹ഑ྀ͢Δ
148. 148 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ
149. 149 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Amazon S3 Chunk Cache
150. 150 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ Storageো֐࣌ʹ΋ϩάΛݕࡧ͢ΔͨΊͷ৚݅ • Ingester͕࠷௿Ұͭ͸݈ࡏͰ͋Δ͜ͱ • ݕࡧ݁ՌΛCache͔IngesterͷσʔλͰΧόʔͰ͖Δ͜ͱ • Cacheʹͳ͍࣌ؒൣғΛΫΤϦʹࢦఆ͠ͳ͍͜ͱ
151. 151 ·ͱΊ
152. 152 ·ͱΊ LokiͷίΞͰ͋Δॻ͖ࠐΈͱಡΈࠐΈϓϩηεΛৄղ • ಈ࡞ݪཧ͕Θ͔ͬͨ͜ͱͰɺτϥϒϧγϡʔςΟϯά΍νϡ ʔχϯά͕Մೳʹ • Ͳ͜ͰͲͷΤϯίʔσΟϯάͰσʔλΛ͔࣋ͭΛ೺Ѳ͢Δ͜ ͱͰΩϟύγςΟϓϥϯχϯά͕Մೳʹ • ো֐࣌ͷڍಈΛ೺Ѳ͢Δ͜ͱͰద੾ͳ४උ͕ݕ౼Մೳʹ
153. 153 ·ͱΊ ఻͑ΒΕͳ͔ͬͨ͜ͱ • ϩά͔ΒͷϝτϦΫεੜ੒΍ΞϥʔςΟϯά • ϩάͷϦςϯγϣϯ؅ཧʹ͍ͭͯ • LokiࣗମͷϞχλϦϯάʹ͍ͭͯ • ֤ίϯϙʔωϯτͷΩϟύγςΟ؅ཧʹ͍ͭͯ • Out of order entry໰୊΁ͷରࡦʹ͍ͭͯ
154. 154 ผ్ຊΛॻ͖·͢
155. 155 Twitter: @taisho6339 ࣭͝໰౳͸
156. 156 THANK YOU