$30 off During Our Annual Pro Sale. View Details »

Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

CloudNative Days Tokyo 2021での登壇資料です
https://event.cloudnativedays.jp/cndt2021/talks/1252

LINE Developers
PRO

November 04, 2021
Tweet

More Decks by LINE Developers

Other Decks in Technology

Transcript

  1. 1
    Grafana LokiͰߏங͢Δ
    େن໛ϩάϞχλϦϯάج൫
    CNDT2021
    LINEגࣜձࣾ Hiroki Sakamoto / @taisho6339

    View Slide

  2. 2
    ࣗݾ঺հ
    - Title:
    Senior Software Engineer@LINE Corp
    - Role:
    Private Cloud։ൃ૊৫ͷSRE
    - Mission
    Private CloudΛԣஅͨ͠৴པվળ
    - Interest:
    Kubernetes, ෼ࢄγεςϜ, ےτϨ, OSS׆ಈ
    - Twitter: @taisho6339

    View Slide

  3. like Prometheus but for logs
    ● ϩάͷอଘͱݕࡧػೳ
    ● ϩάϕʔεͷΞϥʔςΟϯάػೳ
    ● ϩάϕʔεͷϝτϦΫε࡞੒ػೳ
    ● ϚϧνςφϯτͷDefault Support
    3
    Lokiͱ͸Կ͔ʁ

    View Slide

  4. 4
    Lokiͱ͸Կ͔ʁ

    View Slide

  5. 5
    Lokiͱ͸Կ͔ʁ

    View Slide

  6. 6
    ҆͘େ༰ྔͷϩάΛอଘՄೳ

    View Slide

  7. 7
    Private Cloud “Verda” in LINE
    is based on OpenStack. since 2016~
    FaaS
    PaaS
    IaaS
    NAT
    LB
    Bare
    metal

    View Slide

  8. 8
    Private Cloud “Verda” in LINE
    Virtual Machines
    74000+ 30000+ 4000+
    Physical
    Machines
    Hypervisors

    View Slide

  9. 9
    20 TB / day application logs

    View Slide

  10. 10
    Loki is suitable for us!

    View Slide

  11. 11
    Lokiͷ೉͠͞
    Loki͸ϚΠΫϩαʔϏε
    ● ֤ίϯϙʔωϯτɺ֤Ωϟογϡͷ࢓૊Έͱ໾ׂ͕ෆ໌ྎ
    ● ϩάσʔλ͸Ͳ͜ͰͲΜͳܗࣜͰͲͷ͘Β͍อ࣋͞ΕΔ͔ෆ໌ྎ
    ● ετϨʔδো֐࣌͸Ͳ͏͍͏ڍಈʹͳΔͷ͔ෆ໌ྎ
    ● ຊ൪Ͱӡ༻͢ΔͳΒԿΛߟྀ͠ͳ͍ͱ͍͚ͳ͍ͷ͔͕ෆ໌ྎ

    View Slide

  12. 12
    ຊηογϣϯͷΰʔϧ
    ࠃ಺Ͱ࠷΋ৄࡉʹղઆ͢Δ͜ͱΛ໨ࢦ͠·͢
    ● ମܥతʹLokiͷίϯϙʔωϯτͷ໾ׂͱ࢓૊ΈΛ஌Δ
    ● τϥϒϧ࣌ʹ΋ݪҼͷಛఆ͕ਝ଎ʹͰ͖ΔΑ͏ʹͳΔ
    ● ࣗྗͰΩϟύγςΟ؅ཧɺύϑΥʔϚϯενϡʔχϯάͰ͖ΔΑ
    ͏ʹͳΔ

    View Slide

  13. 13
    ຊηογϣϯ૝ఆߏ੒
    Loki Version: v2.3.0
    Cache: Memcached
    Chunk Storage: AWS S3
    Index Storage: AWS S3 + BoltDB Shipper

    View Slide

  14. 14
    1) ϩάͷॻ͖ࠐΈϓϩηεΛ஌Δ
    2) ϩάͷಡΈࠐΈϓϩηεΛ஌Δ
    3) ো֐࣌ͷڍಈΛ஌Δ
    ຊηογϣϯͷRoadmap

    View Slide

  15. 15
    1) ϩάͷॻ͖ࠐΈϓϩηε

    View Slide

  16. 16
    Overview

    View Slide

  17. 17
    ॻ͖ࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)

    View Slide

  18. 18
    ॻ͖ࠐΈϓϩηε Overview
    Amazon S3
    Chunk Cache Index Write Cache
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)
    • Fluentd΍PromtailͳͲͷɺϩάૹ৴Client
    • ϩάΛಡΈऔΓLokiͷΤϯυϙΠϯτ΁ૹ৴͢Δ

    View Slide

  19. 19
    Amazon S3
    Chunk Cache Index Write Cache
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)
    • ϦΫΤετͷόϦσʔγϣϯΛߦ͏
    • ద੾ͳIngester΁ϧʔςΟϯά͢Δ
    ॻ͖ࠐΈϓϩηε Overview

    View Slide

  20. 20
    Amazon S3
    Chunk Cache Index Write Cache
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)
    ϩάΛ࣮ࡍʹετϨʔδʹอଘ͢Δ
    Ұఆ࣌ؒόοϑΝϦϯά͔ͯ͠Βετ
    Ϩʔδʹอଘ
    ॻ͖ࠐΈϓϩηε Overview

    View Slide

  21. 21
    Amazon S3
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)
    • S3ʹϩάΛӬଓԽ
    • MemcachedʹϩάͷΩϟογϡΛอଘ
    Chunk Cache
    ॻ͖ࠐΈϓϩηε Overview

    View Slide

  22. 22
    Client͔ΒDistributor΁ͷૹ৴

    View Slide

  23. 23
    Client -> Distributor
    Clients
    (Promtail, Fluentd)
    Distributor
    HTTP Headers
    X-Scope-OrgID :
    TenantIDΛRequest Headerʹهࡌ

    View Slide

  24. 24
    Client -> Distributor
    Clients
    (Promtail, Fluentd)
    Distributor
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    {service=“keystone”, hostname=“host1”} 00:00:03 keystone log body
    {service=“keystone”, hostname=“host1”} 00:00:04 keystone log body
    Loki΁ૹΔϩάσʔλߏ଄

    View Slide

  25. 25
    Client -> Distributor
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    Stream Log Body
    TS

    View Slide

  26. 26
    Client -> Distributor
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    ϩά͸͍͔ͭ͘ͷϥϕϧΛ࣋ͭɻ
    TenantIDͱϥϕϧͷ૊Έ߹Θͤͷ
    ҰͭҰͭΛɺʮStreamʯͱݺͿ
    Stream Log Body
    TS

    View Slide

  27. 27
    DistributorͰͷόϦσʔγϣϯ
    • ϥϕϧͷܗࣜ͸ਖ਼͍͔͠ʁ
    • Rate limitΛ௒͑ͳ͍͔ʁ
    Clients
    (Promtail, Fluentd)
    Distributor

    View Slide

  28. 28
    Distributor
    DistributorͰͷόϦσʔγϣϯ
    Distributor
    Distributor
    Distributor
    Distributorಉ࢜ͰΫϥελϦϯά
    • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ
    • ingestion_rate_strategy͕global
    ͳΒΫϥελશମͰingestion
    rateΛ੍ޚ͢Δ

    View Slide

  29. 29
    Distributor
    DistributorͰͷόϦσʔγϣϯ
    Distributor
    Distributor
    Distributor
    DistributorશମͰόϦσʔγϣϯ
    ͢ΔͨΊͷΫϥελϦϯά

    View Slide

  30. 30
    Distributor͔ΒIngester΁ͷૹ৴

    View Slide

  31. 31
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    ݕࡧͷͨΊʹ(ޙड़)
    ෳ਺ͷIngesterʹ
    ৑௕Խͯ͠ϩάΛૹΔ

    View Slide

  32. 32
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    Ingesterಉ࢜ͰΫϥελϦϯά
    • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ
    • Consistent HashͷRingʹͳͬͯ
    ͍Δ

    View Slide

  33. 33
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    FNV1-32bitͰHash஋Λੜ੒
    tenantID + {service=“keystone”, hostname=“host1”}
    a6965cd7

    View Slide

  34. 34
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    a6965cd7
    Consistent Hashʹج͍ͮͯɺ
    ࢉग़ͨ͠Hash஋ʹରԠ͢ΔIngester
    Λreplication factor෼ཁٻ

    View Slide

  35. 35
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    ؼ͖ͬͯͨෳ਺ͷIngesterʹɺ
    ಉ͡ϩάΛಉ࣌ૹ৴

    View Slide

  36. 36
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    OK
    OK
    Fail

    View Slide

  37. 37
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    ա൒਺͕OKͳΒ੒ޭ
    OK
    OK
    Fail

    View Slide

  38. 38
    IngesterͷRequest Handling

    View Slide

  39. 39
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    {service=“keystone”, hostname=“host1”}
    Ingester

    View Slide

  40. 40
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    Stream1 chunks
    StreamͰGrouping͞Εɺ
    Chunkͱ͍͏ܗࣜͰAppend
    Ingester

    View Slide

  41. 41
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    Stream1 chunks
    WAL Segment
    Ingester
    WALʹه࿥

    View Slide

  42. 42
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    Stream1 chunks
    WAL Segment
    Ingester
    OK

    View Slide

  43. 43
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    Stream1 chunks
    WAL Segment
    Ingester
    ΋͠Stream಺Ͱ࠷ޙʹड͚औͬͨϩάͷ࣌ؒΑΓ
    લͷ࣌ؒͷϩά͕དྷͨ৔߹͸ϦΫΤετΛࣦഊͤ͞Δ
    Out of order entry error

    View Slide

  44. 44
    ChunkόοϑΝϦϯά

    View Slide

  45. 45
    IngesterͷChunkόοϑΝϦϯά
    • Ұఆ਺ͷϩάΛStream͝ͱʹ
    ʮChunkʯʹ·ͱΊΔ
    • Chunk͸ϝϞϦ্ʹอଘ͞ΕΔ
    • HeadɺBlocksͱ͍͏഑ྻΛอ࣋
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk

    View Slide

  46. 46
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    {service=“keystone”, hostname=“host1”}

    View Slide

  47. 47
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Log
    Append
    ड͚औͬͨϩά͸Ұ୴Head΁
    Append͞ΕΔ

    View Slide

  48. 48
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Log Log
    Append
    1 block size෼ཷ·Δ·Ͱ܁Γฦ͢

    View Slide

  49. 49
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Log Log Log Log Log
    Append
    1 block size෼ཷ·Δ·Ͱ܁Γฦ͢

    View Slide

  50. 50
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Headʹ௥Ճ
    Blockʹ௥Ճ
    ҰఆαΠζཷ·ͬͨΒ
    ઃఆͨ͠ܗࣜͰѹॖ͠ɺ
    blocksʹ௥Ճ͢Δ

    View Slide

  51. 51
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Headʹ௥Ճ
    Block͕Ұఆ਺ཷ·ͬͨΒ
    Read Only ModeʹͳΓɺFlush Queue΁
    (Default 10 blocks, target chunk sizeͰࢦఆ)

    View Slide

  52. 52
    Ingester͔ΒStorage΁Flush

    View Slide

  53. 53
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Chunk Cache
    Disk
    సஔIndex

    View Slide

  54. 54
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    ৚݅Λຬͨ͢ChunkΛݕ஌
    • ࢦఆαΠζʹ౸ୡ
    • ࠷ޙͷߋ৽͔Βchunk idle periodܦա
    • max_chunk_ageܦա

    View Slide

  55. 55
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Disk
    సஔΠϯσοΫε
    Flush Queue΁Enqueue

    View Slide

  56. 56
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk
    Chunk
    Goroutine Goroutine
    Amazon S3
    Disk
    Enqueue͞Εͨ΋ͷΛFlush
    1. ChunkΛS3΁อଘ
    2. Chunk Cache΁อଘ(Write Through)
    3. సஔIndexΛϩʔΧϧBoltDBʹอଘ
    Chunk Cache
    సஔIndex

    View Slide

  57. 57
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk
    Goroutine Goroutine
    Amazon S3
    Disk
    Ingester΁ͷϦΫΤετ͸
    ෳ੡͞Ε͍ͯΔͷͰɺ
    طʹCacheʹೖ͍ͬͯΔChunk͸
    Storage΁ͷॻ͖ࠐΈ͕ൃੜ͠ͳ
    ͍Α͏ʹ੍ޚ͍ͯ͠Δ Chunk
    Chunk Cache
    సஔIndex

    View Slide

  58. 58
    Cacheͷෛՙ෼ࢄ
    Chunk Cache
    Ingesters
    Chunk Cache
    Chunk Cache
    Chunk1 Key
    Chunk2 Key
    Chunk3 Key
    ઃఆʹΑͬͯConsistent HashʹΑΓɺ
    ChunkͷKeyΛݩʹ෼ࢄͯ͠อଘͰ͖Δ

    View Slide

  59. 59
    Write Ahead Log

    View Slide

  60. 60
    ChunkͷFlush(࠶ܝ)
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε

    View Slide

  61. 61
    IngesterͷChunkͷ࣋ͪํ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    Flush͞ΕΔલʹϓϩηε͕ࢭ·Δͱ
    Chunk͕شൃ͢Δ

    View Slide

  62. 62
    WALͷҙٛ
    ӬଓԽͯ͠Memoryͷشൃʹඋ͑Δ

    View Slide

  63. 63
    LokiͷWALͷಛ௃
    ● ϩάड৴࣌ʹɺMemoryͱWAL྆ํʹॻ͖ࠐΉ
    ● WALॻ͖ࠐΈ͕ࣦഊͯ͠΋ॲཧΛࣦഊͤ͞ͳ͍
    ● Ingesterͷϓϩηεىಈ࣌ʹWAL͔Βͷ෮چॲཧ͕ೖΔ
    ● Ұఆظؒ͝ͱʹෆཁͳWAL͸ύʔδ͞ΕΔ

    View Slide

  64. 64
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Log Entry
    ϩάΛૹ৴

    View Slide

  65. 65
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1
    Append
    SegmentϑΝΠϧʹ
    rawσʔλͷ··௥ه͞ΕΔ

    View Slide

  66. 66
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1
    Create!
    1ϑΝΠϧ͕େ͖͘ͳͬͯ͘Δͱ
    ผͷSegmentϑΝΠϧΛ࡞੒
    Segment2

    View Slide

  67. 67
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine Goroutine
    Disk
    Segment1 Segment2
    Ұఆظؒ͝ͱʹ
    ෆཁͳSegmentϑΝΠϧͷ
    ύʔδॲཧ͕ೖΔ

    View Slide

  68. 68
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1 Segment2 Segment3
    Create!
    SegmentΛҰͭਐΊΔ

    View Slide

  69. 69
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1 Segment2 Segment3 Checkpoint1
    IngesterͷະFlush ChunkΛ
    Checkpointͱݺ͹ΕΔ
    εφοϓγϣοτͱͯ͠อଘ

    View Slide

  70. 70
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1 Segment2 Segment3 Checkpoint1
    MemoryChunkߏ଄ͷ··อଘ͢Δ
    ͷͰblock͸ѹॖܗࣜɺ
    headঢ়ଶͷϩά͸ඇѹॖͱͳΔ

    View Slide

  71. 71
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment3 Checkpoint1
    ࠷৽ͷCheckpointͱSegmentΛ࢒
    ͯ͢͠΂ͯ࡟আ͢Δ

    View Slide

  72. 72
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment3 Checkpoint1
    ͦͷ࣌఺ͰFlush͞Ε͍ͯͳ͍
    ͢΂ͯͷϩάؚ͕·Ε͍ͯΔ

    View Slide

  73. 73
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Tenant2
    Goroutine
    Disk
    Segment3 Checkpoint1
    Ingesterͷىಈ࣌ʹ͸
    Disk͔Β
    SegmentͱCheckpointΛಡΈऔΔ

    View Slide

  74. 74
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Tenant2
    Goroutine
    Disk
    Segment3 Checkpoint1
    Stream1
    Stream2
    Stream1
    Stream2
    Memory಺ʹ෮ݩ͢Δ

    View Slide

  75. 75
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Tenant2
    Goroutine
    Disk
    Segment3 Checkpoint1
    Stream1
    Stream2
    Stream1
    Stream2
    ෮ݩ͕׬ྃ͢Δ·Ͱ͸ϓϩηεΛىಈ͠ͳ͍

    View Slide

  76. 76
    Ingester্ͷσʔλͱEncodeܗࣜ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Disk
    Segment1 Segment2 Segment3 Checkpoint1
    1block sizeҎԼͷraw + ѹॖ
    per Chunk
    raw
    Memory Chunkͱಉ༷

    View Slide

  77. 77
    2) ϩάͷಡΈࠐΈϓϩηε

    View Slide

  78. 78
    Overview

    View Slide

  79. 79
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    Queriers
    Query Result
    Cache
    Ingesters

    View Slide

  80. 80
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ϦΫΤετΛड෇͚Δ

    View Slide

  81. 81
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ड͚औͬͨΫΤϦΛ࣌ؒͳͲͰ
    ෼ׂͯ͠ΩϡʔΠϯά͢Δ

    View Slide

  82. 82
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ෳ਺ͷQuerier͕Ωϡʔ͔ΒQuery
    Λड͚औΓϋϯυϦϯά͢Δ
    Queriers

    View Slide

  83. 83
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ͢΂ͯͷIngesterʹରͯ͠ɺ
    MemoryChunk͔ΒQueryʹMatch
    ͢Δ΋ͷΛཁٻ
    Queriers
    Ingesters

    View Slide

  84. 84
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    QueryʹରԠ͢ΔసஔIndexΛऔಘ͢Δ
    ͜ͷͱ͖CacheʹಁաతʹΞΫηε
    Queriers

    View Slide

  85. 85
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    Cacheʹଘࡏ͠ͳ͍৔߹͸ɺ
    ϩʔΧϧͷBoltDB͔ΒMatch͢ΔIndexΛऔಘ
    ͦͯ݁͠ՌΛCacheʹอଘ͢Δ(snappy)
    Queriers
    BoltDB

    View Slide

  86. 86
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    సஔIndex͔Βର৅ͷChunkΛׂΓग़͢
    Queriers

    View Slide

  87. 87
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ChunkΛऔಘ͢Δ
    Cacheʹͳ͍΋ͷ͸Storage͔Βऔಘ͠ɺ
    Cacheʹอଘ͢Δ
    Queriers

    View Slide

  88. 88
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend ͢΂ͯͷQuerier͔Βͷ݁ՌΛड͚औΓɺ
    ू໿ɺιʔτɺॏෳഉআΛ࣮ࢪ
    Queriers

    View Slide

  89. 89
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ݁ՌΛQuery Result Cache΁อଘ͢Δ
    Queriers
    Query Result
    Cache

    View Slide

  90. 90
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    Ϩεϙϯεฦ٫
    Queriers
    Query Result
    Cache

    View Slide

  91. 91
    సஔIndex͔Βର৅Chunkͷબఆ

    View Slide

  92. 92
    సஔIndexͷ໨త
    S3͔ΒChunkΛ࠷খ࿑ྗͰऔಘ͢Δ͜ͱ

    View Slide

  93. 93
    సஔIndex͔Βର৅ChunkͷߜΓࠐΈ
    1.LabelͷKeyͱValueͷ૊Έ߹Θ͔ͤΒStreamͷID
    (Series ID)Λऔಘ͢Δ
    2.Series IDͱ࣌ؒൣғ͔ΒChunkͷKeyΛऔಘ͢Δ
    3.ChunkͷKey͔ΒS3্ɺMemcached্ͷύεΛׂ
    Γग़͠ɺ௚઀ChunkΛDownload

    View Slide

  94. 94
    SeriesID
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    key, valueͷ૊Έ߹Θͤͷsha256
    9ac2adda49e899b312a9abb895656b1ab26c9858fd500f2ae3983d5309b39363/
    ϩά
    SeriesID

    View Slide

  95. 95
    Chunk Key
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    Tenant1/a6965cd7:Chunk։࢝࣌ؒ:Chunkऴྃ࣌ؒ
    key, valueͷ૊Έ߹ΘͤͷHash஋
    a6965cd7
    ϩά
    FingerPrint
    Chunk Key

    View Slide

  96. Hash Value Range Value Value
    TenantID + LabelName Hash(Label Value) + SeriesID Label Value
    సஔIndexͷߏ଄Πϝʔδ
    Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index

    View Slide

  97. Hash Value Range Value Value
    TenantID + LabelName Hash(Label Value) + SeriesID Label Value
    Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index
    Hash + RangeͰϢχʔΫߦΛಛఆ͢Δ
    సஔIndexͷߏ଄Πϝʔδ

    View Slide

  98. Hash Value Range Value Value
    TenantID + LabelName Hash(Label Value) + SeriesID Label Value
    Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index
    సஔIndexͷߏ଄Πϝʔδ
    Range Value͸ൣғݕࡧɺιʔτʹར༻Ͱ͖Δ

    View Slide

  99. Hash Value
    (TenantID + Label Name)
    Range Value
    (Hash(Label Value) + SeriesID)
    Value
    (Label Value)
    Tenant1:service abc680ab:c79abadeff keystone
    Tenant1:host cfe960ab:bcfe12ea hostname1
    Tenant1:type can860ab:c79abadeff api
    Tenant1:service cdc680ab:c79abadeff nova
    Tenant1:host bee960ab:bcfe12ea hostname21
    Tenant1:type abd860ab:c79abadeff scheduler
    ςʔϒϧΠϝʔδ
    సஔIndexͷߏ଄Πϝʔδ

    View Slide

  100. ͜ͷIndex͸
    LabelͷKeyͱValueͷύλʔϯ෼͚ͩ࡞ΒΕΔ
    ΧʔσΟφϦςΟͷߴ͍ϥϕϧ͸
    ͜ͷIndex͕େྔʹ࡞ΒΕΔ͜ͱʹͳΔ
    సஔIndexͷߏ଄Πϝʔδ

    View Slide

  101. SeriesID͔ΒChunk KeyΛҾ͘Index
    సஔIndexͷߏ଄Πϝʔδ
    Hash Value Range Value Value
    TenantID + SeriesID Chunkͷ։࢝࣌ؒ + Chunk Key nil

    View Slide

  102. 102
    Index͔ΒͷChunk KeyׂΓग़͠
    {service=“keystone”, hostname=“host1”} |= “level=ERROR”
    LogQL
    ϥϕϧϚον෦ ϑΟϧλ෦

    View Slide

  103. 103
    Index͔ΒͷChunk KeyׂΓग़͠
    {service=“keystone”, hostname=“host1”} |= “level=ERROR”
    LogQL
    ϥϕϧϚον෦ ϑΟϧλ෦
    IndexΛ࢖͏ͷ͸͜ͷ෦෼ͷධՁ

    View Slide

  104. 104
    Index͔ΒͷChunk KeyׂΓग़͠
    {service=“keystone”, hostname=“host1”}
    {service=“keystone”} {hostname=“host1”}
    ෼ׂ

    View Slide

  105. 105
    Index͔ΒͷChunk KeyׂΓग़͠
    {service=“keystone”, hostname=“host1”}
    {service=“keystone”} {hostname=“host1”}
    ͦΕͧΕͰϚονϯά৚݅
    ͔ΒసஔIndexΛऔಘ
    సஔIndex సஔIndex

    View Slide

  106. Hash Value
    (TenantID + Label Name)
    Range Value
    (Hash(Label Value) + SeriesID) Value
    Tenant1:service abc680ab:c79abadeff keystone
    Tenant1:host cfe960ab:bcfe12ea hostname1
    Tenant1:type can860ab:c79abadeff api
    Tenant1:service cdc680ab:c79abadeff nova
    Tenant1:host bee960ab:bcfe12ea hostname21
    Tenant1:type abd860ab:c79abadeff scheduler
    Index͔ΒͷChunk KeyׂΓग़͠
    {service=“keystone”}
    ώοτ͢Δͷ͸͜ͷϨίʔυ

    View Slide

  107. Hash Value
    (TenantID + Label Name)
    Range Value
    (Hash(Label Value) + SeriesID) Value
    Tenant1:service abc680ab:c79abadeff keystone
    Tenant1:host cfe960ab:bcfe12ea hostname1
    Tenant1:type can860ab:c79abadeff api
    Tenant1:service cdc680ab:c79abadeff nova
    Tenant1:host bee960ab:bcfe12ea hostname21
    Tenant1:type abd860ab:c79abadeff scheduler
    Index͔ΒͷChunk KeyׂΓग़͠
    {service=“keystone”}
    ର৅SeriesID

    View Slide

  108. 108
    Index͔ΒͷChunk KeyׂΓग़͠
    {service=“keystone”, hostname=“host1”}
    {service=“keystone”} {hostname=“host1”}
    Index͔ΒSeriesIDΛநग़
    ྆ํʹڞ௨͢Δ΋ͷ͚ͩ࠾༻
    SeriesIDs
    సஔIndex సஔIndex

    View Slide

  109. 109
    Index͔ΒͷChunk KeyׂΓग़͠
    {service=“keystone”, hostname=“host1”}
    {service=“keystone”} {hostname=“host1”}
    SeriesIDͱ࣌ؒൣғ͔Β
    Chunk KeyΛநग़
    SeriesIDs
    సஔIndex
    Chunk Keys
    సஔIndex సஔIndex

    View Slide

  110. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min
    Hash Value
    (TenantID + SeriesID)
    Range Value
    (Chunk։࢝࣌ؒ + Chunk Key) Value
    Tenant1:c79abadeff 1635252768:chunk1 nil
    Tenant1:c79abadeff 1635256368:chunk2 nil
    Tenant1:c79abadeff 1635260768:chunk3 nil
    Index͔ΒͷChunk KeyׂΓग़͠

    View Slide

  111. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min
    Hash Value
    (TenantID + SeriesID)
    Range Value
    (Chunk։࢝࣌ؒ + Chunk Key) Value
    Tenant1:c79abadeff 1635252768:chunk1 nil
    Tenant1:c79abadeff 1635256368:chunk2 nil
    Tenant1:c79abadeff 1635260768:chunk3 nil
    Index͔ΒͷChunk KeyׂΓग़͠
    ର৅ChunkͷRecord

    View Slide

  112. 112
    Ϩεϙϯεͷੜ੒ͱฦ٫

    View Slide

  113. 113
    Ϩεϙϯεͷੜ੒
    Chunk Keys
    Lazy Chunks
    ॳظঢ়ଶͰ͸࣮ମΛ࣋ͨͣɺ
    ಡΈࠐΈ໋ྩ͕͞ΕͨλΠϛϯάͰετϨʔδ(Ωϟογϡ)
    ʹChunkΛऔΓʹߦ͘Lazy ChunkΛੜ੒

    View Slide

  114. 114
    Ϩεϙϯεͷੜ੒
    Chunk Keys
    Lazy Chunks
    Ingester͔ΒͷChunks
    Iterator
    Ingester͔ΒͷChunkΛ͋Θͤͯɺ
    IteratorΛੜ੒

    View Slide

  115. 115
    Ϩεϙϯεͷੜ੒
    Chunk Keys
    Lazy Chunks
    Ingester͔ΒͷChunks
    Iterator
    Response
    Ϩεϙϯε͕نఆ݅਺ʹୡ͢Δ·Ͱɺ
    IteratorΛಡΈࠐΜͰ͍͘
    ϩάͷFilter৚݅͸͜͜ͰධՁ͞ΕΔ
    |= “level=ERROR”

    View Slide

  116. 116
    Ϩεϙϯεͷੜ੒
    Chunk Keys
    Lazy Chunks
    Ingester͔ΒͷChunks
    Iterator
    Response
    LazyChunkΛಡΈࠐΉ৔߹͸ɺ
    Ωϟογϡ΁ɺͳ͚Ε͹Storage΁
    ChunkΛ໰͍߹ΘͤΔ
    Amazon S3
    Chunk Cache

    View Slide

  117. 117
    Ϩεϙϯεͷੜ੒
    Chunk Keys
    Lazy Chunks
    Ingester͔ΒͷChunks
    Iterator
    Response
    Storage͔ΒಡΈࠐΜͩ৔߹͸ɺ
    औಘޙʹCacheʹอଘ͢Δ
    Amazon S3
    Chunk Cache

    View Slide

  118. 118
    Query Sharding

    View Slide

  119. 119
    Queryͷ෼ׂઓུ
    1. ࣌ؒ͝ͱʹ෼ׂ͢Δ
    ● 1࣌ؒ෼ͷϩάΛݕࡧ͢Δ৔߹ɺ15෼Ͱ෼ׂ͢ΔઃఆͳΒ4
    ͭͷΫΤϦʹ෼ղ͞Ε࣮ͯߦ͞ΕΔ
    2. సஔIndexΛSharding͢Δ
    ● ͋Β͔͡ΊసஔIndexʹShard൪߸Λ͍Ε͓͖ͯɺ
    QueryFrontend͕QueryΛ෼ׂ͠ɺͦΕͧΕʹShard൪߸Λ
    ૠೖͯ͠QuerierʹΘͨ͢

    View Slide

  120. Hash Value Range Value Value
    TenantID + LabelName Shard Number +
    Hash(Label Value) + SeriesID Label Value
    సஔIndex΁ͷShard൪߸ຒΊࠐΈ
    Shard Number = SeriesID % shard count

    View Slide

  121. 121
    సஔIndex΁ͷShard൪߸ຒΊࠐΈ
    {service=“keystone”, hostname=“host1”}
    {service=“keystone”}
    {service=“keystone”,hostname=“host2”}
    1
    12
    16
    Shard Number
    Stream

    View Slide

  122. 122
    Shard൪߸ʹΑΔQuery෼ׂ
    {service=“keystone”}
    |= “level=ERROR”
    LogQL
    Querier for shard 1
    Querier for shard 12
    Querier for shard 16
    QueryΛShardͰ෼ׂ

    View Slide

  123. 123
    Shard൪߸ʹΑΔQuery෼ׂ
    {service=“keystone”, hostname=“host1”}
    {service=“keystone”}
    {service=“keystone”,hostname=“host2”}
    1
    12
    16
    {service=“keystone”}
    |= “level=ERROR”
    LogQL
    Querier for shard 1
    Querier for shard 12
    Querier for shard 16
    Chunk Keys
    Chunk Keys
    Chunk Keys
    औΕΔChunk͕shardͰ෼ׂ͞ΕΔ

    View Slide

  124. 124
    Shard൪߸ʹΑΔQuery෼ׂ
    {service=“keystone”, hostname=“host1”}
    {service=“keystone”}
    {service=“keystone”,hostname=“host2”}
    1
    12
    16
    {service=“keystone”}
    |= “level=ERROR”
    LogQL
    Querier for shard 1
    Querier for shard 12
    Querier for shard 16
    Chunk Keys
    Chunk Keys
    Chunk Keys
    |= “level=ERROR”ͷfilterॲཧΛ෼ׂॲཧͰ͖Δ

    View Slide

  125. 125
    BoltDB ShipperʹΑΔIndex؅ཧ

    View Slide

  126. 126
    BoltDB Shipper
    Ingester
    Shipper
    Disk
    Querier
    Shipper
    Disk
    Index 1 Index 2 Index 1 Index 2
    Amazon S3

    View Slide

  127. 127
    BoltDB Shipper - Ingester side
    Ingester
    Shipper
    Disk
    Querier
    Shipper
    Disk
    Index 1 Index 2 Index 1 Index 2
    Amazon S3
    Ұఆ࣌ؒ͝ͱʹϩʔΧϧDiskʹ͋ΔIndex
    ΛS3΁Ξοϓϩʔυ͢Δ
    Ξοϓϩʔυޙʹ࡟আ͢Δ

    View Slide

  128. 128
    BoltDB Shipper - Querier side
    Ingester
    Shipper
    Disk
    Querier
    Shipper
    Disk
    Index 1 Index 2 Index 1 Index 2
    Amazon S3
    • ىಈ࣌ʹS3ʹ͋ΔIndexΛDownload
    • Query࣌ʹ଍Γͳ͍Index͸S3͔Β౎౓
    μ΢ϯϩʔυ
    • Ұఆ࣌ؒ͝ͱʹ࠷ऴ࢖༻͔Β
    CacheTTLܦաͨ͠IndexΛ࡟আ

    View Slide

  129. 129
    BoltDB Shipper
    Ingester
    Shipper
    Disk
    Querier
    Shipper
    Disk
    Index 1 Index 2 Index 1 Index 2
    Amazon S3
    Ingester, Querier͸ϩʔΧϧͰIndexΛѻ͍ɺ
    Shipper͕ඇಉظͰIndexΛStorageͱಉظ͢Δ

    View Slide

  130. 130
    ֤ίϯϙʔωϯτͷ໾ׂ·ͱΊ

    View Slide

  131. 131
    Name ໾ׂ λΠϓ σʔλͷ࣋ͪԽ ΫϥελϦϯά༗ແ
    Distributor όϦσʔγϣϯͱIngester΁ͷϧʔςΟϯά Stateless ༗
    Ingester σʔλͷόοϑΝϦϯάͱFlush Stateful
    Memory:
    Chunks(raw + ѹॖ)
    Disk:
    WAL(raw + ѹॖ)
    సஔIndex(ѹॖ)

    Query
    Frontend ΫΤϦͷ෼ׂɺΩϡʔ੍ޚ Stateless ແ
    Querier ΫΤϦͷ࣮ߦ Stateful Disk:
    సஔIndexͷCache(ѹॖ) ແ
    Chunk Cache ChunkͷΩϟογϡ Stateful Memory: Chunks(ѹॖ) ༗(ΫϥΠΞϯταΠυ)
    Index Read
    Cache IndexͷRead༻Ωϟογϡ Stateful Memory:
    సஔIndex(Snappy)
    ༗(ΫϥΠΞϯταΠυ)
    Index Write
    Cache
    ಉ͡Indexͷॻ͖ࠐΈ͕ෳ਺ൃੜ͠ͳ͍Α͏ʹ
    ͢ΔͨΊͷ੍ޚ༻Ωϟογϡ
    (BoltDB ShipperͰ͸ෆཁ)
    Stateful
    Memory:
    Chunk Key(raw)
    ༗(ΫϥΠΞϯταΠυ)
    Query Result
    Cache
    ΫΤϦͷ݁ՌͷΩϟογϡ Stateful Memory:
    Query Result(raw)
    ༗(ΫϥΠΞϯταΠυ)

    View Slide

  132. 132
    3) ো֐࣌ͷڍಈΛ஌Δ

    View Slide

  133. 133
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ

    View Slide

  134. 134
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    S3ো֐࣌ʹඋ͑Δ

    View Slide

  135. 135
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    Segment1 Checkpoint1
    ͕ࣦ͜͜ഊ
    సஔIndex

    View Slide

  136. 136
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    Segment1 Checkpoint1
    Flush͸ඇಉظͳͷͰ
    ॻ͖ࠐΈࣗମ͸ࣦഊ͠ͳ͍

    View Slide

  137. 137
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    Segment1 Checkpoint1
    ͕ࣦ͜͜ഊ
    MemoryͱDiskʹͨ·Γଓ͚Δ

    View Slide

  138. 138
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    Segment1 Checkpoint1
    ͕ࣦ͜͜ഊ
    ઌʹMemory͕͋;ΕɺOOM

    View Slide

  139. 139
    WALͷ࢓૊Έ(࠶ܝ)
    Ingester
    Tenant1
    Memory
    Tenant2
    Goroutine
    Disk
    Segment3 Checkpoint1
    Ingesterͷىಈ࣌ʹ͸
    Disk͔Β
    SegmentͱCheckpointΛಡΈऔΔ

    View Slide

  140. 140
    WALͷ࢓૊Έ(࠶ܝ)
    Ingester
    Tenant1
    Memory
    Tenant2
    Goroutine
    Disk
    Segment3 Checkpoint1
    Stream1
    Stream2
    Stream1
    Stream2
    Memory಺ʹ෮ݩ͢Δ

    View Slide

  141. 141
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2 Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    Segment1 Checkpoint1
    WAL͸࣮࣭MemoryͷSnapshot
    ϩʔυ੒ޭͯ͠΋͍ۙ͏ͪʹOOM
    ϩʔυࣦഊͨ͠Βͦ΋ͦ΋ىಈ͠ͳ͍

    View Slide

  142. 142
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2 Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    Segment1 Checkpoint1
    ϘτϧωοΫ͸IngesterͷMemory

    View Slide

  143. 143
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Disk
    Segment1 Segment2 Segment3 Checkpoint1
    1block sizeҎԼͷraw + ѹॖ
    per Chunk
    raw
    Memory Chunkͱಉ༷
    Ingester্ͷσʔλͱEncodeܗࣜ(࠶ܝ)

    View Slide

  144. 144
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    ରࡦ1. ଱͍͑ͨ࣌ؒ෼ͷMemoryΛੵΉ
    ● ࣌ؒ͋ͨΓͷϩάྔ / ѹॖൺ * ࣌ؒ * replication factor
    ※ѹॖൺ͸ฐ؀ڥͰ͸gzipѹॖͰ10~18ഒͷѹॖൺ

    View Slide

  145. 145
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    1೔ͷྲྀྔ10TBͷ؀ڥͰ1࣌ؒ଱͑Δ(ฐࣾͷ1Ϧʔδϣϯ)
    1000 / 24 = 41.6 GB / hour
    ● Replication Factor = 1
    ● Ingester * 11୆
    ● Memory: 4GiB
    ● Disk: 8GiB(ϚʔδϯΛऔͬͯMemoryͷ2ഒ)
    ● Chunk Cache * 14୆
    ● Memory: 3GiB

    View Slide

  146. 146
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    Replication Factor͸1Ͱ͍͍ͷ͔ʁ
    ● Flush͞ΕΔ·ͰʹIngesterϓϩηε͕μ΢ϯ͢Δͱͦͷؒ
    ͚ͩͦʹ͋ͬͨϩά͸ܽଛ͢Δ
    ͋Δఔ౓ׂΓ੾ΓΛ͢Δ
    ● WAL͕͋ΔͷͰ࠶ىಈޙʹ͙͢ʹ෮چͰ͖Δ
    ● ࡉ͔͍ܽଛΑΓো֐࣌ʹՔಇܧଓͰ͖ΔՄೳੑΛߴΊΔํ
    ޲ʹৼΔ

    View Slide

  147. 147
    ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
    ରࡦ2. ো֐࣌͸Ұ࣌తʹWALΛແޮʹͯ͠ىಈ͢Δ
    ● WALϩʔυ͕૸Βͳ͍ͷͰɺMemory͔Β͋;ΕΔྔཷ·͍ͬͯ
    ͯ΋ϓϩηεىಈͰ͖ΔΑ͏ʹͳΔ
    ● ࠶༗ޮʹ͢Δͱ͖ʹϩά͕شൃ͠ͳ͍Α͏ɺreplication factorɺ
    update strategyʹ഑ྀ͢Δ

    View Slide

  148. 148
    ಡΈऔΓ࣌ͷ଱ো֐ઃܭ

    View Slide

  149. 149
    ಡΈऔΓ࣌ͷ଱ো֐ઃܭ
    Chunk Keys
    Lazy Chunks
    Ingester͔ΒͷChunks
    Iterator
    Response
    Amazon S3
    Chunk Cache

    View Slide

  150. 150
    ಡΈऔΓ࣌ͷ଱ো֐ઃܭ
    Storageো֐࣌ʹ΋ϩάΛݕࡧ͢ΔͨΊͷ৚݅
    ● Ingester͕࠷௿Ұͭ͸݈ࡏͰ͋Δ͜ͱ
    ● ݕࡧ݁ՌΛCache͔IngesterͷσʔλͰΧόʔͰ͖Δ͜ͱ
    ● Cacheʹͳ͍࣌ؒൣғΛΫΤϦʹࢦఆ͠ͳ͍͜ͱ

    View Slide

  151. 151
    ·ͱΊ

    View Slide

  152. 152
    ·ͱΊ
    LokiͷίΞͰ͋Δॻ͖ࠐΈͱಡΈࠐΈϓϩηεΛৄղ
    ● ಈ࡞ݪཧ͕Θ͔ͬͨ͜ͱͰɺτϥϒϧγϡʔςΟϯά΍νϡ
    ʔχϯά͕Մೳʹ
    ● Ͳ͜ͰͲͷΤϯίʔσΟϯάͰσʔλΛ͔࣋ͭΛ೺Ѳ͢Δ͜
    ͱͰΩϟύγςΟϓϥϯχϯά͕Մೳʹ
    ● ো֐࣌ͷڍಈΛ೺Ѳ͢Δ͜ͱͰద੾ͳ४උ͕ݕ౼Մೳʹ

    View Slide

  153. 153
    ·ͱΊ
    ఻͑ΒΕͳ͔ͬͨ͜ͱ
    ● ϩά͔ΒͷϝτϦΫεੜ੒΍ΞϥʔςΟϯά
    ● ϩάͷϦςϯγϣϯ؅ཧʹ͍ͭͯ
    ● LokiࣗମͷϞχλϦϯάʹ͍ͭͯ
    ● ֤ίϯϙʔωϯτͷΩϟύγςΟ؅ཧʹ͍ͭͯ
    ● Out of order entry໰୊΁ͷରࡦʹ͍ͭͯ

    View Slide

  154. 154
    ผ్ຊΛॻ͖·͢

    View Slide

  155. 155
    Twitter: @taisho6339
    ࣭͝໰౳͸

    View Slide

  156. 156
    THANK YOU

    View Slide