Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

CloudNative Days Tokyo 2021での登壇資料です
https://event.cloudnativedays.jp/cndt2021/talks/1252

Transcript

1. 1
   Grafana LokiͰߏங͢Δ
   େن໛ϩάϞχλϦϯάج൫
   CNDT2021
   LINEגࣜձࣾ Hiroki Sakamoto / @taisho6339
   

   View Slide

2. 2
   ࣗݾ঺հ
   - Title:
   Senior Software Engineer@LINE Corp
   - Role:
   Private Cloud։ൃ૊৫ͷSRE
   - Mission
   Private CloudΛԣஅͨ͠৴པվળ
   - Interest:
   Kubernetes, ෼ࢄγεςϜ, ےτϨ, OSS׆ಈ
   - Twitter: @taisho6339
   

   View Slide

3. like Prometheus but for logs
   ● ϩάͷอଘͱݕࡧػೳ
   ● ϩάϕʔεͷΞϥʔςΟϯάػೳ
   ● ϩάϕʔεͷϝτϦΫε࡞੒ػೳ
   ● ϚϧνςφϯτͷDefault Support
   3
   Lokiͱ͸Կ͔ʁ
   

   View Slide

4. 4
   Lokiͱ͸Կ͔ʁ
   

   View Slide

5. 5
   Lokiͱ͸Կ͔ʁ
   

   View Slide

6. 6
   ҆͘େ༰ྔͷϩάΛอଘՄೳ
   

   View Slide

7. 7
   Private Cloud “Verda” in LINE
   is based on OpenStack. since 2016~
   FaaS
   PaaS
   IaaS
   NAT
   LB
   Bare
   metal
   

   View Slide

8. 8
   Private Cloud “Verda” in LINE
   Virtual Machines
   74000+ 30000+ 4000+
   Physical
   Machines
   Hypervisors
   

   View Slide

9. 9
   20 TB / day application logs
   

   View Slide

10. 10
    Loki is suitable for us!
    

    View Slide

11. 11
    Lokiͷ೉͠͞
    Loki͸ϚΠΫϩαʔϏε
    ● ֤ίϯϙʔωϯτɺ֤Ωϟογϡͷ࢓૊Έͱ໾ׂ͕ෆ໌ྎ
    ● ϩάσʔλ͸Ͳ͜ͰͲΜͳܗࣜͰͲͷ͘Β͍อ࣋͞ΕΔ͔ෆ໌ྎ
    ● ετϨʔδো֐࣌͸Ͳ͏͍͏ڍಈʹͳΔͷ͔ෆ໌ྎ
    ● ຊ൪Ͱӡ༻͢ΔͳΒԿΛߟྀ͠ͳ͍ͱ͍͚ͳ͍ͷ͔͕ෆ໌ྎ
    

    View Slide

12. 12
    ຊηογϣϯͷΰʔϧ
    ࠃ಺Ͱ࠷΋ৄࡉʹղઆ͢Δ͜ͱΛ໨ࢦ͠·͢
    ● ମܥతʹLokiͷίϯϙʔωϯτͷ໾ׂͱ࢓૊ΈΛ஌Δ
    ● τϥϒϧ࣌ʹ΋ݪҼͷಛఆ͕ਝ଎ʹͰ͖ΔΑ͏ʹͳΔ
    ● ࣗྗͰΩϟύγςΟ؅ཧɺύϑΥʔϚϯενϡʔχϯάͰ͖ΔΑ
    ͏ʹͳΔ
    

    View Slide

13. 13
    ຊηογϣϯ૝ఆߏ੒
    Loki Version: v2.3.0
    Cache: Memcached
    Chunk Storage: AWS S3
    Index Storage: AWS S3 + BoltDB Shipper
    

    View Slide

14. 14
    1) ϩάͷॻ͖ࠐΈϓϩηεΛ஌Δ
    2) ϩάͷಡΈࠐΈϓϩηεΛ஌Δ
    3) ো֐࣌ͷڍಈΛ஌Δ
    ຊηογϣϯͷRoadmap
    

    View Slide

15. 15
    1) ϩάͷॻ͖ࠐΈϓϩηε
    

    View Slide

16. 16
    Overview
    

    View Slide

17. 17
    ॻ͖ࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)
    

    View Slide

18. 18
    ॻ͖ࠐΈϓϩηε Overview
    Amazon S3
    Chunk Cache Index Write Cache
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)
    • Fluentd΍PromtailͳͲͷɺϩάૹ৴Client
    • ϩάΛಡΈऔΓLokiͷΤϯυϙΠϯτ΁ૹ৴͢Δ
    

    View Slide

19. 19
    Amazon S3
    Chunk Cache Index Write Cache
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)
    • ϦΫΤετͷόϦσʔγϣϯΛߦ͏
    • ద੾ͳIngester΁ϧʔςΟϯά͢Δ
    ॻ͖ࠐΈϓϩηε Overview
    

    View Slide

20. 20
    Amazon S3
    Chunk Cache Index Write Cache
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)
    ϩάΛ࣮ࡍʹετϨʔδʹอଘ͢Δ
    Ұఆ࣌ؒόοϑΝϦϯά͔ͯ͠Βετ
    Ϩʔδʹอଘ
    ॻ͖ࠐΈϓϩηε Overview
    

    View Slide

21. 21
    Amazon S3
    Loki
    Clients Storage
    Distributor Ingesters
    Clients
    (Promtail, Fluentd)
    • S3ʹϩάΛӬଓԽ
    • MemcachedʹϩάͷΩϟογϡΛอଘ
    Chunk Cache
    ॻ͖ࠐΈϓϩηε Overview
    

    View Slide

22. 22
    Client͔ΒDistributor΁ͷૹ৴
    

    View Slide

23. 23
    Client -> Distributor
    Clients
    (Promtail, Fluentd)
    Distributor
    HTTP Headers
    X-Scope-OrgID :
    TenantIDΛRequest Headerʹهࡌ
    

    View Slide

24. 24
    Client -> Distributor
    Clients
    (Promtail, Fluentd)
    Distributor
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    {service=“keystone”, hostname=“host1”} 00:00:03 keystone log body
    {service=“keystone”, hostname=“host1”} 00:00:04 keystone log body
    Loki΁ૹΔϩάσʔλߏ଄
    

    View Slide

25. 25
    Client -> Distributor
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    Stream Log Body
    TS
    

    View Slide

26. 26
    Client -> Distributor
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    ϩά͸͍͔ͭ͘ͷϥϕϧΛ࣋ͭɻ
    TenantIDͱϥϕϧͷ૊Έ߹Θͤͷ
    ҰͭҰͭΛɺʮStreamʯͱݺͿ
    Stream Log Body
    TS
    

    View Slide

27. 27
    DistributorͰͷόϦσʔγϣϯ
    • ϥϕϧͷܗࣜ͸ਖ਼͍͔͠ʁ
    • Rate limitΛ௒͑ͳ͍͔ʁ
    Clients
    (Promtail, Fluentd)
    Distributor
    

    View Slide

28. 28
    Distributor
    DistributorͰͷόϦσʔγϣϯ
    Distributor
    Distributor
    Distributor
    Distributorಉ࢜ͰΫϥελϦϯά
    • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ
    • ingestion_rate_strategy͕global
    ͳΒΫϥελશମͰingestion
    rateΛ੍ޚ͢Δ
    

    View Slide

29. 29
    Distributor
    DistributorͰͷόϦσʔγϣϯ
    Distributor
    Distributor
    Distributor
    DistributorશମͰόϦσʔγϣϯ
    ͢ΔͨΊͷΫϥελϦϯά
    

    View Slide

30. 30
    Distributor͔ΒIngester΁ͷૹ৴
    

    View Slide

31. 31
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    ݕࡧͷͨΊʹ(ޙड़)
    ෳ਺ͷIngesterʹ
    ৑௕Խͯ͠ϩάΛૹΔ
    

    View Slide

32. 32
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    Ingesterಉ࢜ͰΫϥελϦϯά
    • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ
    • Consistent HashͷRingʹͳͬͯ
    ͍Δ
    

    View Slide

33. 33
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    FNV1-32bitͰHash஋Λੜ੒
    tenantID + {service=“keystone”, hostname=“host1”}
    a6965cd7
    

    View Slide

34. 34
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    a6965cd7
    Consistent Hashʹج͍ͮͯɺ
    ࢉग़ͨ͠Hash஋ʹରԠ͢ΔIngester
    Λreplication factor෼ཁٻ
    

    View Slide

35. 35
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    ؼ͖ͬͯͨෳ਺ͷIngesterʹɺ
    ಉ͡ϩάΛಉ࣌ૹ৴
    

    View Slide

36. 36
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    OK
    OK
    Fail
    

    View Slide

37. 37
    Distributor -> Ingesters
    Ingester
    Ingesters
    Ingester
    Ingester
    Distributor
    ա൒਺͕OKͳΒ੒ޭ
    OK
    OK
    Fail
    

    View Slide

38. 38
    IngesterͷRequest Handling
    

    View Slide

39. 39
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    {service=“keystone”, hostname=“host1”}
    Ingester
    

    View Slide

40. 40
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    Stream1 chunks
    StreamͰGrouping͞Εɺ
    Chunkͱ͍͏ܗࣜͰAppend
    Ingester
    

    View Slide

41. 41
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    Stream1 chunks
    WAL Segment
    Ingester
    WALʹه࿥
    

    View Slide

42. 42
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    Stream1 chunks
    WAL Segment
    Ingester
    OK
    

    View Slide

43. 43
    IngesterͷRequest Handling
    Memory
    Tenant1
    Disk
    Tenant2
    Stream1 chunks
    WAL Segment
    Ingester
    ΋͠Stream಺Ͱ࠷ޙʹड͚औͬͨϩάͷ࣌ؒΑΓ
    લͷ࣌ؒͷϩά͕དྷͨ৔߹͸ϦΫΤετΛࣦഊͤ͞Δ
    Out of order entry error
    

    View Slide

44. 44
    ChunkόοϑΝϦϯά
    

    View Slide

45. 45
    IngesterͷChunkόοϑΝϦϯά
    • Ұఆ਺ͷϩάΛStream͝ͱʹ
    ʮChunkʯʹ·ͱΊΔ
    • Chunk͸ϝϞϦ্ʹอଘ͞ΕΔ
    • HeadɺBlocksͱ͍͏഑ྻΛอ࣋
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    

    View Slide

46. 46
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    {service=“keystone”, hostname=“host1”}
    

    View Slide

47. 47
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Log
    Append
    ड͚औͬͨϩά͸Ұ୴Head΁
    Append͞ΕΔ
    

    View Slide

48. 48
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Log Log
    Append
    1 block size෼ཷ·Δ·Ͱ܁Γฦ͢
    

    View Slide

49. 49
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Log Log Log Log Log
    Append
    1 block size෼ཷ·Δ·Ͱ܁Γฦ͢
    

    View Slide

50. 50
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Headʹ௥Ճ
    Blockʹ௥Ճ
    ҰఆαΠζཷ·ͬͨΒ
    ઃఆͨ͠ܗࣜͰѹॖ͠ɺ
    blocksʹ௥Ճ͢Δ
    

    View Slide

51. 51
    IngesterͷChunkόοϑΝϦϯά
    Head
    Blocks
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    compressed block
    MemoryChunk
    Headʹ௥Ճ
    Block͕Ұఆ਺ཷ·ͬͨΒ
    Read Only ModeʹͳΓɺFlush Queue΁
    (Default 10 blocks, target chunk sizeͰࢦఆ)
    

    View Slide

52. 52
    Ingester͔ΒStorage΁Flush
    

    View Slide

53. 53
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Chunk Cache
    Disk
    సஔIndex
    

    View Slide

54. 54
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    ৚݅Λຬͨ͢ChunkΛݕ஌
    • ࢦఆαΠζʹ౸ୡ
    • ࠷ޙͷߋ৽͔Βchunk idle periodܦա
    • max_chunk_ageܦա
    

    View Slide

55. 55
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Disk
    సஔΠϯσοΫε
    Flush Queue΁Enqueue
    

    View Slide

56. 56
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk
    Chunk
    Goroutine Goroutine
    Amazon S3
    Disk
    Enqueue͞Εͨ΋ͷΛFlush
    1. ChunkΛS3΁อଘ
    2. Chunk Cache΁อଘ(Write Through)
    3. సஔIndexΛϩʔΧϧBoltDBʹอଘ
    Chunk Cache
    సஔIndex
    

    View Slide

57. 57
    ChunkͷFlush
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk
    Goroutine Goroutine
    Amazon S3
    Disk
    Ingester΁ͷϦΫΤετ͸
    ෳ੡͞Ε͍ͯΔͷͰɺ
    طʹCacheʹೖ͍ͬͯΔChunk͸
    Storage΁ͷॻ͖ࠐΈ͕ൃੜ͠ͳ
    ͍Α͏ʹ੍ޚ͍ͯ͠Δ Chunk
    Chunk Cache
    సஔIndex
    

    View Slide

58. 58
    Cacheͷෛՙ෼ࢄ
    Chunk Cache
    Ingesters
    Chunk Cache
    Chunk Cache
    Chunk1 Key
    Chunk2 Key
    Chunk3 Key
    ઃఆʹΑͬͯConsistent HashʹΑΓɺ
    ChunkͷKeyΛݩʹ෼ࢄͯ͠อଘͰ͖Δ
    

    View Slide

59. 59
    Write Ahead Log
    

    View Slide

60. 60
    ChunkͷFlush(࠶ܝ)
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    

    View Slide

61. 61
    IngesterͷChunkͷ࣋ͪํ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Chunk Chunk
    Goroutine Goroutine
    Amazon S3
    Index Write
    Cache
    Chunk
    Cache
    Disk
    సஔΠϯσοΫε
    Flush͞ΕΔલʹϓϩηε͕ࢭ·Δͱ
    Chunk͕شൃ͢Δ
    

    View Slide

62. 62
    WALͷҙٛ
    ӬଓԽͯ͠Memoryͷشൃʹඋ͑Δ
    

    View Slide

63. 63
    LokiͷWALͷಛ௃
    ● ϩάड৴࣌ʹɺMemoryͱWAL྆ํʹॻ͖ࠐΉ
    ● WALॻ͖ࠐΈ͕ࣦഊͯ͠΋ॲཧΛࣦഊͤ͞ͳ͍
    ● Ingesterͷϓϩηεىಈ࣌ʹWAL͔Βͷ෮چॲཧ͕ೖΔ
    ● Ұఆظؒ͝ͱʹෆཁͳWAL͸ύʔδ͞ΕΔ
    

    View Slide

64. 64
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Log Entry
    ϩάΛૹ৴
    

    View Slide

65. 65
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1
    Append
    SegmentϑΝΠϧʹ
    rawσʔλͷ··௥ه͞ΕΔ
    

    View Slide

66. 66
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1
    Create!
    1ϑΝΠϧ͕େ͖͘ͳͬͯ͘Δͱ
    ผͷSegmentϑΝΠϧΛ࡞੒
    Segment2
    

    View Slide

67. 67
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine Goroutine
    Disk
    Segment1 Segment2
    Ұఆظؒ͝ͱʹ
    ෆཁͳSegmentϑΝΠϧͷ
    ύʔδॲཧ͕ೖΔ
    

    View Slide

68. 68
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1 Segment2 Segment3
    Create!
    SegmentΛҰͭਐΊΔ
    

    View Slide

69. 69
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1 Segment2 Segment3 Checkpoint1
    IngesterͷະFlush ChunkΛ
    Checkpointͱݺ͹ΕΔ
    εφοϓγϣοτͱͯ͠อଘ
    

    View Slide

70. 70
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment1 Segment2 Segment3 Checkpoint1
    MemoryChunkߏ଄ͷ··อଘ͢Δ
    ͷͰblock͸ѹॖܗࣜɺ
    headঢ়ଶͷϩά͸ඇѹॖͱͳΔ
    

    View Slide

71. 71
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment3 Checkpoint1
    ࠷৽ͷCheckpointͱSegmentΛ࢒
    ͯ͢͠΂ͯ࡟আ͢Δ
    

    View Slide

72. 72
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Goroutine
    Disk
    Segment3 Checkpoint1
    ͦͷ࣌఺ͰFlush͞Ε͍ͯͳ͍
    ͢΂ͯͷϩάؚ͕·Ε͍ͯΔ
    

    View Slide

73. 73
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Tenant2
    Goroutine
    Disk
    Segment3 Checkpoint1
    Ingesterͷىಈ࣌ʹ͸
    Disk͔Β
    SegmentͱCheckpointΛಡΈऔΔ
    

    View Slide

74. 74
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Tenant2
    Goroutine
    Disk
    Segment3 Checkpoint1
    Stream1
    Stream2
    Stream1
    Stream2
    Memory಺ʹ෮ݩ͢Δ
    

    View Slide

75. 75
    WALͷ࢓૊Έ
    Ingester
    Tenant1
    Memory
    Tenant2
    Goroutine
    Disk
    Segment3 Checkpoint1
    Stream1
    Stream2
    Stream1
    Stream2
    ෮ݩ͕׬ྃ͢Δ·Ͱ͸ϓϩηεΛىಈ͠ͳ͍
    

    View Slide

76. 76
    Ingester্ͷσʔλͱEncodeܗࣜ
    Ingester
    Tenant1
    Memory
    Stream1
    Stream2
    Tenant2
    Stream1
    Stream2
    Disk
    Segment1 Segment2 Segment3 Checkpoint1
    1block sizeҎԼͷraw + ѹॖ
    per Chunk
    raw
    Memory Chunkͱಉ༷
    

    View Slide

77. 77
    2) ϩάͷಡΈࠐΈϓϩηε
    

    View Slide

78. 78
    Overview
    

    View Slide

79. 79
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    Queriers
    Query Result
    Cache
    Ingesters
    

    View Slide

80. 80
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ϦΫΤετΛड෇͚Δ
    

    View Slide

81. 81
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ड͚औͬͨΫΤϦΛ࣌ؒͳͲͰ
    ෼ׂͯ͠ΩϡʔΠϯά͢Δ
    

    View Slide

82. 82
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ෳ਺ͷQuerier͕Ωϡʔ͔ΒQuery
    Λड͚औΓϋϯυϦϯά͢Δ
    Queriers
    

    View Slide

83. 83
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ͢΂ͯͷIngesterʹରͯ͠ɺ
    MemoryChunk͔ΒQueryʹMatch
    ͢Δ΋ͷΛཁٻ
    Queriers
    Ingesters
    

    View Slide

84. 84
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    QueryʹରԠ͢ΔసஔIndexΛऔಘ͢Δ
    ͜ͷͱ͖CacheʹಁաతʹΞΫηε
    Queriers
    

    View Slide

85. 85
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    Cacheʹଘࡏ͠ͳ͍৔߹͸ɺ
    ϩʔΧϧͷBoltDB͔ΒMatch͢ΔIndexΛऔಘ
    ͦͯ݁͠ՌΛCacheʹอଘ͢Δ(snappy)
    Queriers
    BoltDB
    

    View Slide

86. 86
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    సஔIndex͔Βର৅ͷChunkΛׂΓग़͢
    Queriers
    

    View Slide

87. 87
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ChunkΛऔಘ͢Δ
    Cacheʹͳ͍΋ͷ͸Storage͔Βऔಘ͠ɺ
    Cacheʹอଘ͢Δ
    Queriers
    

    View Slide

88. 88
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend ͢΂ͯͷQuerier͔Βͷ݁ՌΛड͚औΓɺ
    ू໿ɺιʔτɺॏෳഉআΛ࣮ࢪ
    Queriers
    

    View Slide

89. 89
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    ݁ՌΛQuery Result Cache΁อଘ͢Δ
    Queriers
    Query Result
    Cache
    

    View Slide

90. 90
    ಡΈࠐΈϓϩηεͷొ৔ਓ෺
    Amazon S3
    Chunk Cache Index Read Cache
    Loki Storage
    Query Frontend
    Ϩεϙϯεฦ٫
    Queriers
    Query Result
    Cache
    

    View Slide

91. 91
    సஔIndex͔Βର৅Chunkͷબఆ
    

    View Slide

92. 92
    సஔIndexͷ໨త
    S3͔ΒChunkΛ࠷খ࿑ྗͰऔಘ͢Δ͜ͱ
    

    View Slide

93. 93
    సஔIndex͔Βର৅ChunkͷߜΓࠐΈ
    1.LabelͷKeyͱValueͷ૊Έ߹Θ͔ͤΒStreamͷID
    (Series ID)Λऔಘ͢Δ
    2.Series IDͱ࣌ؒൣғ͔ΒChunkͷKeyΛऔಘ͢Δ
    3.ChunkͷKey͔ΒS3্ɺMemcached্ͷύεΛׂ
    Γग़͠ɺ௚઀ChunkΛDownload
    

    View Slide

94. 94
    SeriesID
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    key, valueͷ૊Έ߹Θͤͷsha256
    9ac2adda49e899b312a9abb895656b1ab26c9858fd500f2ae3983d5309b39363/
    ϩά
    SeriesID
    

    View Slide

95. 95
    Chunk Key
    {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body
    Tenant1/a6965cd7:Chunk։࢝࣌ؒ:Chunkऴྃ࣌ؒ
    key, valueͷ૊Έ߹ΘͤͷHash஋
    a6965cd7
    ϩά
    FingerPrint
    Chunk Key
    

    View Slide

96. Hash Value Range Value Value
    TenantID + LabelName Hash(Label Value) + SeriesID Label Value
    సஔIndexͷߏ଄Πϝʔδ
    Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index
    

    View Slide

97. Hash Value Range Value Value
    TenantID + LabelName Hash(Label Value) + SeriesID Label Value
    Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index
    Hash + RangeͰϢχʔΫߦΛಛఆ͢Δ
    సஔIndexͷߏ଄Πϝʔδ
    

    View Slide

98. Hash Value Range Value Value
    TenantID + LabelName Hash(Label Value) + SeriesID Label Value
    Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index
    సஔIndexͷߏ଄Πϝʔδ
    Range Value͸ൣғݕࡧɺιʔτʹར༻Ͱ͖Δ
    

    View Slide

99. Hash Value
    (TenantID + Label Name)
    Range Value
    (Hash(Label Value) + SeriesID)
    Value
    (Label Value)
    Tenant1:service abc680ab:c79abadeff keystone
    Tenant1:host cfe960ab:bcfe12ea hostname1
    Tenant1:type can860ab:c79abadeff api
    Tenant1:service cdc680ab:c79abadeff nova
    Tenant1:host bee960ab:bcfe12ea hostname21
    Tenant1:type abd860ab:c79abadeff scheduler
    ςʔϒϧΠϝʔδ
    సஔIndexͷߏ଄Πϝʔδ
    

    View Slide

100. ͜ͷIndex͸
     LabelͷKeyͱValueͷύλʔϯ෼͚ͩ࡞ΒΕΔ
     ΧʔσΟφϦςΟͷߴ͍ϥϕϧ͸
     ͜ͷIndex͕େྔʹ࡞ΒΕΔ͜ͱʹͳΔ
     సஔIndexͷߏ଄Πϝʔδ
     

     View Slide

101. SeriesID͔ΒChunk KeyΛҾ͘Index
     సஔIndexͷߏ଄Πϝʔδ
     Hash Value Range Value Value
     TenantID + SeriesID Chunkͷ։࢝࣌ؒ + Chunk Key nil
     

     View Slide

102. 102
     Index͔ΒͷChunk KeyׂΓग़͠
     {service=“keystone”, hostname=“host1”} |= “level=ERROR”
     LogQL
     ϥϕϧϚον෦ ϑΟϧλ෦
     

     View Slide

103. 103
     Index͔ΒͷChunk KeyׂΓग़͠
     {service=“keystone”, hostname=“host1”} |= “level=ERROR”
     LogQL
     ϥϕϧϚον෦ ϑΟϧλ෦
     IndexΛ࢖͏ͷ͸͜ͷ෦෼ͷධՁ
     

     View Slide

104. 104
     Index͔ΒͷChunk KeyׂΓग़͠
     {service=“keystone”, hostname=“host1”}
     {service=“keystone”} {hostname=“host1”}
     ෼ׂ
     

     View Slide

105. 105
     Index͔ΒͷChunk KeyׂΓग़͠
     {service=“keystone”, hostname=“host1”}
     {service=“keystone”} {hostname=“host1”}
     ͦΕͧΕͰϚονϯά৚݅
     ͔ΒసஔIndexΛऔಘ
     సஔIndex సஔIndex
     

     View Slide

106. Hash Value
     (TenantID + Label Name)
     Range Value
     (Hash(Label Value) + SeriesID) Value
     Tenant1:service abc680ab:c79abadeff keystone
     Tenant1:host cfe960ab:bcfe12ea hostname1
     Tenant1:type can860ab:c79abadeff api
     Tenant1:service cdc680ab:c79abadeff nova
     Tenant1:host bee960ab:bcfe12ea hostname21
     Tenant1:type abd860ab:c79abadeff scheduler
     Index͔ΒͷChunk KeyׂΓग़͠
     {service=“keystone”}
     ώοτ͢Δͷ͸͜ͷϨίʔυ
     

     View Slide

107. Hash Value
     (TenantID + Label Name)
     Range Value
     (Hash(Label Value) + SeriesID) Value
     Tenant1:service abc680ab:c79abadeff keystone
     Tenant1:host cfe960ab:bcfe12ea hostname1
     Tenant1:type can860ab:c79abadeff api
     Tenant1:service cdc680ab:c79abadeff nova
     Tenant1:host bee960ab:bcfe12ea hostname21
     Tenant1:type abd860ab:c79abadeff scheduler
     Index͔ΒͷChunk KeyׂΓग़͠
     {service=“keystone”}
     ର৅SeriesID
     

     View Slide

108. 108
     Index͔ΒͷChunk KeyׂΓग़͠
     {service=“keystone”, hostname=“host1”}
     {service=“keystone”} {hostname=“host1”}
     Index͔ΒSeriesIDΛநग़
     ྆ํʹڞ௨͢Δ΋ͷ͚ͩ࠾༻
     SeriesIDs
     సஔIndex సஔIndex
     

     View Slide

109. 109
     Index͔ΒͷChunk KeyׂΓग़͠
     {service=“keystone”, hostname=“host1”}
     {service=“keystone”} {hostname=“host1”}
     SeriesIDͱ࣌ؒൣғ͔Β
     Chunk KeyΛநग़
     SeriesIDs
     సஔIndex
     Chunk Keys
     సஔIndex సஔIndex
     

     View Slide

110. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min
     Hash Value
     (TenantID + SeriesID)
     Range Value
     (Chunk։࢝࣌ؒ + Chunk Key) Value
     Tenant1:c79abadeff 1635252768:chunk1 nil
     Tenant1:c79abadeff 1635256368:chunk2 nil
     Tenant1:c79abadeff 1635260768:chunk3 nil
     Index͔ΒͷChunk KeyׂΓग़͠
     

     View Slide

111. SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min
     Hash Value
     (TenantID + SeriesID)
     Range Value
     (Chunk։࢝࣌ؒ + Chunk Key) Value
     Tenant1:c79abadeff 1635252768:chunk1 nil
     Tenant1:c79abadeff 1635256368:chunk2 nil
     Tenant1:c79abadeff 1635260768:chunk3 nil
     Index͔ΒͷChunk KeyׂΓग़͠
     ର৅ChunkͷRecord
     

     View Slide

112. 112
     Ϩεϙϯεͷੜ੒ͱฦ٫
     

     View Slide

113. 113
     Ϩεϙϯεͷੜ੒
     Chunk Keys
     Lazy Chunks
     ॳظঢ়ଶͰ͸࣮ମΛ࣋ͨͣɺ
     ಡΈࠐΈ໋ྩ͕͞ΕͨλΠϛϯάͰετϨʔδ(Ωϟογϡ)
     ʹChunkΛऔΓʹߦ͘Lazy ChunkΛੜ੒
     

     View Slide

114. 114
     Ϩεϙϯεͷੜ੒
     Chunk Keys
     Lazy Chunks
     Ingester͔ΒͷChunks
     Iterator
     Ingester͔ΒͷChunkΛ͋Θͤͯɺ
     IteratorΛੜ੒
     

     View Slide

115. 115
     Ϩεϙϯεͷੜ੒
     Chunk Keys
     Lazy Chunks
     Ingester͔ΒͷChunks
     Iterator
     Response
     Ϩεϙϯε͕نఆ݅਺ʹୡ͢Δ·Ͱɺ
     IteratorΛಡΈࠐΜͰ͍͘
     ϩάͷFilter৚݅͸͜͜ͰධՁ͞ΕΔ
     |= “level=ERROR”
     

     View Slide

116. 116
     Ϩεϙϯεͷੜ੒
     Chunk Keys
     Lazy Chunks
     Ingester͔ΒͷChunks
     Iterator
     Response
     LazyChunkΛಡΈࠐΉ৔߹͸ɺ
     Ωϟογϡ΁ɺͳ͚Ε͹Storage΁
     ChunkΛ໰͍߹ΘͤΔ
     Amazon S3
     Chunk Cache
     

     View Slide

117. 117
     Ϩεϙϯεͷੜ੒
     Chunk Keys
     Lazy Chunks
     Ingester͔ΒͷChunks
     Iterator
     Response
     Storage͔ΒಡΈࠐΜͩ৔߹͸ɺ
     औಘޙʹCacheʹอଘ͢Δ
     Amazon S3
     Chunk Cache
     

     View Slide

118. 118
     Query Sharding
     

     View Slide

119. 119
     Queryͷ෼ׂઓུ
     1. ࣌ؒ͝ͱʹ෼ׂ͢Δ
     ● 1࣌ؒ෼ͷϩάΛݕࡧ͢Δ৔߹ɺ15෼Ͱ෼ׂ͢ΔઃఆͳΒ4
     ͭͷΫΤϦʹ෼ղ͞Ε࣮ͯߦ͞ΕΔ
     2. సஔIndexΛSharding͢Δ
     ● ͋Β͔͡ΊసஔIndexʹShard൪߸Λ͍Ε͓͖ͯɺ
     QueryFrontend͕QueryΛ෼ׂ͠ɺͦΕͧΕʹShard൪߸Λ
     ૠೖͯ͠QuerierʹΘͨ͢
     

     View Slide

120. Hash Value Range Value Value
     TenantID + LabelName Shard Number +
     Hash(Label Value) + SeriesID Label Value
     సஔIndex΁ͷShard൪߸ຒΊࠐΈ
     Shard Number = SeriesID % shard count
     

     View Slide

121. 121
     సஔIndex΁ͷShard൪߸ຒΊࠐΈ
     {service=“keystone”, hostname=“host1”}
     {service=“keystone”}
     {service=“keystone”,hostname=“host2”}
     1
     12
     16
     Shard Number
     Stream
     

     View Slide

122. 122
     Shard൪߸ʹΑΔQuery෼ׂ
     {service=“keystone”}
     |= “level=ERROR”
     LogQL
     Querier for shard 1
     Querier for shard 12
     Querier for shard 16
     QueryΛShardͰ෼ׂ
     

     View Slide

123. 123
     Shard൪߸ʹΑΔQuery෼ׂ
     {service=“keystone”, hostname=“host1”}
     {service=“keystone”}
     {service=“keystone”,hostname=“host2”}
     1
     12
     16
     {service=“keystone”}
     |= “level=ERROR”
     LogQL
     Querier for shard 1
     Querier for shard 12
     Querier for shard 16
     Chunk Keys
     Chunk Keys
     Chunk Keys
     औΕΔChunk͕shardͰ෼ׂ͞ΕΔ
     

     View Slide

124. 124
     Shard൪߸ʹΑΔQuery෼ׂ
     {service=“keystone”, hostname=“host1”}
     {service=“keystone”}
     {service=“keystone”,hostname=“host2”}
     1
     12
     16
     {service=“keystone”}
     |= “level=ERROR”
     LogQL
     Querier for shard 1
     Querier for shard 12
     Querier for shard 16
     Chunk Keys
     Chunk Keys
     Chunk Keys
     |= “level=ERROR”ͷfilterॲཧΛ෼ׂॲཧͰ͖Δ
     

     View Slide

125. 125
     BoltDB ShipperʹΑΔIndex؅ཧ
     

     View Slide

126. 126
     BoltDB Shipper
     Ingester
     Shipper
     Disk
     Querier
     Shipper
     Disk
     Index 1 Index 2 Index 1 Index 2
     Amazon S3
     

     View Slide

127. 127
     BoltDB Shipper - Ingester side
     Ingester
     Shipper
     Disk
     Querier
     Shipper
     Disk
     Index 1 Index 2 Index 1 Index 2
     Amazon S3
     Ұఆ࣌ؒ͝ͱʹϩʔΧϧDiskʹ͋ΔIndex
     ΛS3΁Ξοϓϩʔυ͢Δ
     Ξοϓϩʔυޙʹ࡟আ͢Δ
     

     View Slide

128. 128
     BoltDB Shipper - Querier side
     Ingester
     Shipper
     Disk
     Querier
     Shipper
     Disk
     Index 1 Index 2 Index 1 Index 2
     Amazon S3
     • ىಈ࣌ʹS3ʹ͋ΔIndexΛDownload
     • Query࣌ʹ଍Γͳ͍Index͸S3͔Β౎౓
     μ΢ϯϩʔυ
     • Ұఆ࣌ؒ͝ͱʹ࠷ऴ࢖༻͔Β
     CacheTTLܦաͨ͠IndexΛ࡟আ
     

     View Slide

129. 129
     BoltDB Shipper
     Ingester
     Shipper
     Disk
     Querier
     Shipper
     Disk
     Index 1 Index 2 Index 1 Index 2
     Amazon S3
     Ingester, Querier͸ϩʔΧϧͰIndexΛѻ͍ɺ
     Shipper͕ඇಉظͰIndexΛStorageͱಉظ͢Δ
     

     View Slide

130. 130
     ֤ίϯϙʔωϯτͷ໾ׂ·ͱΊ
     

     View Slide

131. 131
     Name ໾ׂ λΠϓ σʔλͷ࣋ͪԽ ΫϥελϦϯά༗ແ
     Distributor όϦσʔγϣϯͱIngester΁ͷϧʔςΟϯά Stateless ༗
     Ingester σʔλͷόοϑΝϦϯάͱFlush Stateful
     Memory:
     Chunks(raw + ѹॖ)
     Disk:
     WAL(raw + ѹॖ)
     సஔIndex(ѹॖ)
     ༗
     Query
     Frontend ΫΤϦͷ෼ׂɺΩϡʔ੍ޚ Stateless ແ
     Querier ΫΤϦͷ࣮ߦ Stateful Disk:
     సஔIndexͷCache(ѹॖ) ແ
     Chunk Cache ChunkͷΩϟογϡ Stateful Memory: Chunks(ѹॖ) ༗(ΫϥΠΞϯταΠυ)
     Index Read
     Cache IndexͷRead༻Ωϟογϡ Stateful Memory:
     సஔIndex(Snappy)
     ༗(ΫϥΠΞϯταΠυ)
     Index Write
     Cache
     ಉ͡Indexͷॻ͖ࠐΈ͕ෳ਺ൃੜ͠ͳ͍Α͏ʹ
     ͢ΔͨΊͷ੍ޚ༻Ωϟογϡ
     (BoltDB ShipperͰ͸ෆཁ)
     Stateful
     Memory:
     Chunk Key(raw)
     ༗(ΫϥΠΞϯταΠυ)
     Query Result
     Cache
     ΫΤϦͷ݁ՌͷΩϟογϡ Stateful Memory:
     Query Result(raw)
     ༗(ΫϥΠΞϯταΠυ)
     

     View Slide

132. 132
     3) ো֐࣌ͷڍಈΛ஌Δ
     

     View Slide

133. 133
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     

     View Slide

134. 134
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     S3ো֐࣌ʹඋ͑Δ
     

     View Slide

135. 135
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     Ingester
     Tenant1
     Memory
     Stream1
     Stream2
     Tenant2
     Stream1
     Stream2
     Chunk Chunk
     Goroutine Goroutine
     Amazon S3
     Index Write
     Cache
     Chunk
     Cache
     Disk
     Segment1 Checkpoint1
     ͕ࣦ͜͜ഊ
     సஔIndex
     

     View Slide

136. 136
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     Ingester
     Tenant1
     Memory
     Stream1
     Stream2
     Tenant2
     Stream1
     Stream2
     Chunk Chunk
     Goroutine Goroutine
     Amazon S3
     Index Write
     Cache
     Chunk
     Cache
     Disk
     సஔΠϯσοΫε
     Segment1 Checkpoint1
     Flush͸ඇಉظͳͷͰ
     ॻ͖ࠐΈࣗମ͸ࣦഊ͠ͳ͍
     

     View Slide

137. 137
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     Ingester
     Tenant1
     Memory
     Stream1
     Stream2
     Tenant2
     Stream1
     Stream2
     Chunk Chunk
     Goroutine Goroutine
     Amazon S3
     Index Write
     Cache
     Chunk
     Cache
     Disk
     సஔΠϯσοΫε
     Segment1 Checkpoint1
     ͕ࣦ͜͜ഊ
     MemoryͱDiskʹͨ·Γଓ͚Δ
     

     View Slide

138. 138
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     Ingester
     Tenant1
     Memory
     Stream1
     Stream2
     Tenant2
     Stream1
     Stream2
     Chunk Chunk
     Goroutine Goroutine
     Amazon S3
     Index Write
     Cache
     Chunk
     Cache
     Disk
     సஔΠϯσοΫε
     Segment1 Checkpoint1
     ͕ࣦ͜͜ഊ
     ઌʹMemory͕͋;ΕɺOOM
     

     View Slide

139. 139
     WALͷ࢓૊Έ(࠶ܝ)
     Ingester
     Tenant1
     Memory
     Tenant2
     Goroutine
     Disk
     Segment3 Checkpoint1
     Ingesterͷىಈ࣌ʹ͸
     Disk͔Β
     SegmentͱCheckpointΛಡΈऔΔ
     

     View Slide

140. 140
     WALͷ࢓૊Έ(࠶ܝ)
     Ingester
     Tenant1
     Memory
     Tenant2
     Goroutine
     Disk
     Segment3 Checkpoint1
     Stream1
     Stream2
     Stream1
     Stream2
     Memory಺ʹ෮ݩ͢Δ
     

     View Slide

141. 141
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     Ingester
     Tenant1
     Memory
     Stream1
     Stream2
     Tenant2
     Stream1
     Stream2 Amazon S3
     Index Write
     Cache
     Chunk
     Cache
     Disk
     సஔΠϯσοΫε
     Segment1 Checkpoint1
     WAL͸࣮࣭MemoryͷSnapshot
     ϩʔυ੒ޭͯ͠΋͍ۙ͏ͪʹOOM
     ϩʔυࣦഊͨ͠Βͦ΋ͦ΋ىಈ͠ͳ͍
     

     View Slide

142. 142
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     Ingester
     Tenant1
     Memory
     Stream1
     Stream2
     Tenant2
     Stream1
     Stream2 Amazon S3
     Index Write
     Cache
     Chunk
     Cache
     Disk
     సஔΠϯσοΫε
     Segment1 Checkpoint1
     ϘτϧωοΫ͸IngesterͷMemory
     

     View Slide

143. 143
     Ingester
     Tenant1
     Memory
     Stream1
     Stream2
     Tenant2
     Stream1
     Stream2
     Disk
     Segment1 Segment2 Segment3 Checkpoint1
     1block sizeҎԼͷraw + ѹॖ
     per Chunk
     raw
     Memory Chunkͱಉ༷
     Ingester্ͷσʔλͱEncodeܗࣜ(࠶ܝ)
     

     View Slide

144. 144
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     ରࡦ1. ଱͍͑ͨ࣌ؒ෼ͷMemoryΛੵΉ
     ● ࣌ؒ͋ͨΓͷϩάྔ / ѹॖൺ * ࣌ؒ * replication factor
     ※ѹॖൺ͸ฐ؀ڥͰ͸gzipѹॖͰ10~18ഒͷѹॖൺ
     

     View Slide

145. 145
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     1೔ͷྲྀྔ10TBͷ؀ڥͰ1࣌ؒ଱͑Δ(ฐࣾͷ1Ϧʔδϣϯ)
     1000 / 24 = 41.6 GB / hour
     ● Replication Factor = 1
     ● Ingester * 11୆
     ● Memory: 4GiB
     ● Disk: 8GiB(ϚʔδϯΛऔͬͯMemoryͷ2ഒ)
     ● Chunk Cache * 14୆
     ● Memory: 3GiB
     

     View Slide

146. 146
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     Replication Factor͸1Ͱ͍͍ͷ͔ʁ
     ● Flush͞ΕΔ·ͰʹIngesterϓϩηε͕μ΢ϯ͢Δͱͦͷؒ
     ͚ͩͦʹ͋ͬͨϩά͸ܽଛ͢Δ
     ͋Δఔ౓ׂΓ੾ΓΛ͢Δ
     ● WAL͕͋ΔͷͰ࠶ىಈޙʹ͙͢ʹ෮چͰ͖Δ
     ● ࡉ͔͍ܽଛΑΓো֐࣌ʹՔಇܧଓͰ͖ΔՄೳੑΛߴΊΔํ
     ޲ʹৼΔ
     

     View Slide

147. 147
     ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ
     ରࡦ2. ো֐࣌͸Ұ࣌తʹWALΛແޮʹͯ͠ىಈ͢Δ
     ● WALϩʔυ͕૸Βͳ͍ͷͰɺMemory͔Β͋;ΕΔྔཷ·͍ͬͯ
     ͯ΋ϓϩηεىಈͰ͖ΔΑ͏ʹͳΔ
     ● ࠶༗ޮʹ͢Δͱ͖ʹϩά͕شൃ͠ͳ͍Α͏ɺreplication factorɺ
     update strategyʹ഑ྀ͢Δ
     

     View Slide

148. 148
     ಡΈऔΓ࣌ͷ଱ো֐ઃܭ
     

     View Slide

149. 149
     ಡΈऔΓ࣌ͷ଱ো֐ઃܭ
     Chunk Keys
     Lazy Chunks
     Ingester͔ΒͷChunks
     Iterator
     Response
     Amazon S3
     Chunk Cache
     

     View Slide

150. 150
     ಡΈऔΓ࣌ͷ଱ো֐ઃܭ
     Storageো֐࣌ʹ΋ϩάΛݕࡧ͢ΔͨΊͷ৚݅
     ● Ingester͕࠷௿Ұͭ͸݈ࡏͰ͋Δ͜ͱ
     ● ݕࡧ݁ՌΛCache͔IngesterͷσʔλͰΧόʔͰ͖Δ͜ͱ
     ● Cacheʹͳ͍࣌ؒൣғΛΫΤϦʹࢦఆ͠ͳ͍͜ͱ
     

     View Slide

151. 151
     ·ͱΊ
     

     View Slide

152. 152
     ·ͱΊ
     LokiͷίΞͰ͋Δॻ͖ࠐΈͱಡΈࠐΈϓϩηεΛৄղ
     ● ಈ࡞ݪཧ͕Θ͔ͬͨ͜ͱͰɺτϥϒϧγϡʔςΟϯά΍νϡ
     ʔχϯά͕Մೳʹ
     ● Ͳ͜ͰͲͷΤϯίʔσΟϯάͰσʔλΛ͔࣋ͭΛ೺Ѳ͢Δ͜
     ͱͰΩϟύγςΟϓϥϯχϯά͕Մೳʹ
     ● ো֐࣌ͷڍಈΛ೺Ѳ͢Δ͜ͱͰద੾ͳ४උ͕ݕ౼Մೳʹ
     

     View Slide

153. 153
     ·ͱΊ
     ఻͑ΒΕͳ͔ͬͨ͜ͱ
     ● ϩά͔ΒͷϝτϦΫεੜ੒΍ΞϥʔςΟϯά
     ● ϩάͷϦςϯγϣϯ؅ཧʹ͍ͭͯ
     ● LokiࣗମͷϞχλϦϯάʹ͍ͭͯ
     ● ֤ίϯϙʔωϯτͷΩϟύγςΟ؅ཧʹ͍ͭͯ
     ● Out of order entry໰୊΁ͷରࡦʹ͍ͭͯ
     

     View Slide

154. 154
     ผ్ຊΛॻ͖·͢
     

     View Slide

155. 155
     Twitter: @taisho6339
     ࣭͝໰౳͸
     

     View Slide

156. 156
     THANK YOU
     

     View Slide