Grafana Lokiで構築する大規模ログモニタリング基盤 / Grafana Loki Deep Dive

by LINE Developers

Slide 1

Slide 1 text

1 Grafana LokiͰߏங͢Δ େن໛ϩάϞχλϦϯάج൫ CNDT2021 LINEגࣜձࣾ Hiroki Sakamoto / @taisho6339

Slide 2

Slide 2 text

2 ࣗݾ঺հ - Title: Senior Software Engineer@LINE Corp - Role: Private Cloud։ൃ૊৫ͷSRE - Mission Private CloudΛԣஅͨ͠৴པվળ - Interest: Kubernetes, ෼ࢄγεςϜ, ےτϨ, OSS׆ಈ - Twitter: @taisho6339

Slide 3

Slide 3 text

like Prometheus but for logs ● ϩάͷอଘͱݕࡧػೳ ● ϩάϕʔεͷΞϥʔςΟϯάػೳ ● ϩάϕʔεͷϝτϦΫε࡞੒ػೳ ● ϚϧνςφϯτͷDefault Support 3 Lokiͱ͸Կ͔ʁ

Slide 4

Slide 4 text

4 Lokiͱ͸Կ͔ʁ

Slide 5

Slide 5 text

5 Lokiͱ͸Կ͔ʁ

Slide 6

Slide 6 text

6 ҆͘େ༰ྔͷϩάΛอଘՄೳ

Slide 7

Slide 7 text

7 Private Cloud “Verda” in LINE is based on OpenStack. since 2016~ FaaS PaaS IaaS NAT LB Bare metal

Slide 8

Slide 8 text

8 Private Cloud “Verda” in LINE Virtual Machines 74000+ 30000+ 4000+ Physical Machines Hypervisors

Slide 9

Slide 9 text

9 20 TB / day application logs

Slide 10

Slide 10 text

10 Loki is suitable for us!

Slide 11

Slide 11 text

11 Lokiͷ೉͠͞ Loki͸ϚΠΫϩαʔϏε ● ֤ίϯϙʔωϯτɺ֤Ωϟογϡͷ࢓૊Έͱ໾ׂ͕ෆ໌ྎ ● ϩάσʔλ͸Ͳ͜ͰͲΜͳܗࣜͰͲͷ͘Β͍อ࣋͞ΕΔ͔ෆ໌ྎ ● ετϨʔδো֐࣌͸Ͳ͏͍͏ڍಈʹͳΔͷ͔ෆ໌ྎ ● ຊ൪Ͱӡ༻͢ΔͳΒԿΛߟྀ͠ͳ͍ͱ͍͚ͳ͍ͷ͔͕ෆ໌ྎ

Slide 12

Slide 12 text

12 ຊηογϣϯͷΰʔϧ ࠃ಺Ͱ࠷΋ৄࡉʹղઆ͢Δ͜ͱΛ໨ࢦ͠·͢ ● ମܥతʹLokiͷίϯϙʔωϯτͷ໾ׂͱ࢓૊ΈΛ஌Δ ● τϥϒϧ࣌ʹ΋ݪҼͷಛఆ͕ਝ଎ʹͰ͖ΔΑ͏ʹͳΔ ● ࣗྗͰΩϟύγςΟ؅ཧɺύϑΥʔϚϯενϡʔχϯάͰ͖ΔΑ ͏ʹͳΔ

Slide 13

Slide 13 text

13 ຊηογϣϯ૝ఆߏ੒ Loki Version: v2.3.0 Cache: Memcached Chunk Storage: AWS S3 Index Storage: AWS S3 + BoltDB Shipper

Slide 14

Slide 14 text

14 1) ϩάͷॻ͖ࠐΈϓϩηεΛ஌Δ 2) ϩάͷಡΈࠐΈϓϩηεΛ஌Δ 3) ো֐࣌ͷڍಈΛ஌Δ ຊηογϣϯͷRoadmap

Slide 15

Slide 15 text

15 1) ϩάͷॻ͖ࠐΈϓϩηε

Slide 16

Slide 16 text

16 Overview

Slide 17

Slide 17 text

17 ॻ͖ࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd)

Slide 18

Slide 18 text

18 ॻ͖ࠐΈϓϩηε Overview Amazon S3 Chunk Cache Index Write Cache Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) • Fluentd΍PromtailͳͲͷɺϩάૹ৴Client • ϩάΛಡΈऔΓLokiͷΤϯυϙΠϯτ΁ૹ৴͢Δ

Slide 19

Slide 19 text

19 Amazon S3 Chunk Cache Index Write Cache Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) • ϦΫΤετͷόϦσʔγϣϯΛߦ͏ • ద੾ͳIngester΁ϧʔςΟϯά͢Δ ॻ͖ࠐΈϓϩηε Overview

Slide 20

Slide 20 text

20 Amazon S3 Chunk Cache Index Write Cache Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) ϩάΛ࣮ࡍʹετϨʔδʹอଘ͢Δ Ұఆ࣌ؒόοϑΝϦϯά͔ͯ͠Βετ Ϩʔδʹอଘ ॻ͖ࠐΈϓϩηε Overview

Slide 21

Slide 21 text

21 Amazon S3 Loki Clients Storage Distributor Ingesters Clients (Promtail, Fluentd) • S3ʹϩάΛӬଓԽ • MemcachedʹϩάͷΩϟογϡΛอଘ Chunk Cache ॻ͖ࠐΈϓϩηε Overview

Slide 22

Slide 22 text

22 Client͔ΒDistributor΁ͷૹ৴

Slide 23

Slide 23 text

23 Client -> Distributor Clients (Promtail, Fluentd) Distributor HTTP Headers X-Scope-OrgID : TenantIDΛRequest Headerʹهࡌ

Slide 24

Slide 24 text

24 Client -> Distributor Clients (Promtail, Fluentd) Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body {service=“keystone”, hostname=“host1”} 00:00:03 keystone log body {service=“keystone”, hostname=“host1”} 00:00:04 keystone log body Loki΁ૹΔϩάσʔλߏ଄

Slide 25

Slide 25 text

25 Client -> Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body Stream Log Body TS

Slide 26

Slide 26 text

26 Client -> Distributor {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body ϩά͸͍͔ͭ͘ͷϥϕϧΛ࣋ͭɻ TenantIDͱϥϕϧͷ૊Έ߹Θͤͷ ҰͭҰͭΛɺʮStreamʯͱݺͿ Stream Log Body TS

Slide 27

Slide 27 text

27 DistributorͰͷόϦσʔγϣϯ • ϥϕϧͷܗࣜ͸ਖ਼͍͔͠ʁ • Rate limitΛ௒͑ͳ͍͔ʁ Clients (Promtail, Fluentd) Distributor

Slide 28

Slide 28 text

28 Distributor DistributorͰͷόϦσʔγϣϯ Distributor Distributor Distributor Distributorಉ࢜ͰΫϥελϦϯά • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ • ingestion_rate_strategy͕global ͳΒΫϥελશମͰingestion rateΛ੍ޚ͢Δ

Slide 29

Slide 29 text

29 Distributor DistributorͰͷόϦσʔγϣϯ Distributor Distributor Distributor DistributorશମͰόϦσʔγϣϯ ͢ΔͨΊͷΫϥελϦϯά

Slide 30

Slide 30 text

30 Distributor͔ΒIngester΁ͷૹ৴

Slide 31

Slide 31 text

31 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ݕࡧͷͨΊʹ(ޙड़) ෳ਺ͷIngesterʹ ৑௕Խͯ͠ϩάΛૹΔ

Slide 32

Slide 32 text

32 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor Ingesterಉ࢜ͰΫϥελϦϯά • StatusΛޓ͍ʹ૬ޓʹ؂ࢹ • Consistent HashͷRingʹͳͬͯ ͍Δ

Slide 33

Slide 33 text

33 Distributor -> Ingesters Ingester Ingesters Ingester Ingester FNV1-32bitͰHash஋Λੜ੒ tenantID + {service=“keystone”, hostname=“host1”} a6965cd7

Slide 34

Slide 34 text

34 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor a6965cd7 Consistent Hashʹج͍ͮͯɺ ࢉग़ͨ͠Hash஋ʹରԠ͢ΔIngester Λreplication factor෼ཁٻ

Slide 35

Slide 35 text

35 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ؼ͖ͬͯͨෳ਺ͷIngesterʹɺ ಉ͡ϩάΛಉ࣌ૹ৴

Slide 36

Slide 36 text

36 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor OK OK Fail

Slide 37

Slide 37 text

37 Distributor -> Ingesters Ingester Ingesters Ingester Ingester Distributor ա൒਺͕OKͳΒ੒ޭ OK OK Fail

Slide 38

Slide 38 text

38 IngesterͷRequest Handling

Slide 39

Slide 39 text

39 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 {service=“keystone”, hostname=“host1”} Ingester

Slide 40

Slide 40 text

40 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks StreamͰGrouping͞Εɺ Chunkͱ͍͏ܗࣜͰAppend Ingester

Slide 41

Slide 41 text

41 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL Segment Ingester WALʹه࿥

Slide 42

Slide 42 text

42 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL Segment Ingester OK

Slide 43

Slide 43 text

43 IngesterͷRequest Handling Memory Tenant1 Disk Tenant2 Stream1 chunks WAL Segment Ingester ΋͠Stream಺Ͱ࠷ޙʹड͚औͬͨϩάͷ࣌ؒΑΓ લͷ࣌ؒͷϩά͕དྷͨ৔߹͸ϦΫΤετΛࣦഊͤ͞Δ Out of order entry error

Slide 44

Slide 44 text

44 ChunkόοϑΝϦϯά

Slide 45

Slide 45 text

45 IngesterͷChunkόοϑΝϦϯά • Ұఆ਺ͷϩάΛStream͝ͱʹ ʮChunkʯʹ·ͱΊΔ • Chunk͸ϝϞϦ্ʹอଘ͞ΕΔ • HeadɺBlocksͱ͍͏഑ྻΛอ࣋ Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk

Slide 46

Slide 46 text

46 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk {service=“keystone”, hostname=“host1”}

Slide 47

Slide 47 text

47 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk Log Append ड͚औͬͨϩά͸Ұ୴Head΁ Append͞ΕΔ

Slide 48

Slide 48 text

48 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢

Slide 49

Slide 49 text

49 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block MemoryChunk Log Log Log Log Log Append 1 block size෼ཷ·Δ·Ͱ܁Γฦ͢

Slide 50

Slide 50 text

50 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Blockʹ௥Ճ ҰఆαΠζཷ·ͬͨΒ ઃఆͨ͠ܗࣜͰѹॖ͠ɺ blocksʹ௥Ճ͢Δ

Slide 51

Slide 51 text

51 IngesterͷChunkόοϑΝϦϯά Head Blocks compressed block compressed block compressed block compressed block compressed block compressed block MemoryChunk Headʹ௥Ճ Block͕Ұఆ਺ཷ·ͬͨΒ Read Only ModeʹͳΓɺFlush Queue΁ (Default 10 blocks, target chunk sizeͰࢦఆ)

Slide 52

Slide 52 text

52 Ingester͔ΒStorage΁Flush

Slide 53

Slide 53 text

53 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Chunk Cache Disk సஔIndex

Slide 54

Slide 54 text

54 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε ৚݅Λຬͨ͢ChunkΛݕ஌ • ࢦఆαΠζʹ౸ୡ • ࠷ޙͷߋ৽͔Βchunk idle periodܦա • max_chunk_ageܦա

Slide 55

Slide 55 text

55 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Disk సஔΠϯσοΫε Flush Queue΁Enqueue

Slide 56

Slide 56 text

56 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Disk Enqueue͞Εͨ΋ͷΛFlush 1. ChunkΛS3΁อଘ 2. Chunk Cache΁อଘ(Write Through) 3. సஔIndexΛϩʔΧϧBoltDBʹอଘ Chunk Cache సஔIndex

Slide 57

Slide 57 text

57 ChunkͷFlush Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Goroutine Goroutine Amazon S3 Disk Ingester΁ͷϦΫΤετ͸ ෳ੡͞Ε͍ͯΔͷͰɺ طʹCacheʹೖ͍ͬͯΔChunk͸ Storage΁ͷॻ͖ࠐΈ͕ൃੜ͠ͳ ͍Α͏ʹ੍ޚ͍ͯ͠Δ Chunk Chunk Cache సஔIndex

Slide 58

Slide 58 text

58 Cacheͷෛՙ෼ࢄ Chunk Cache Ingesters Chunk Cache Chunk Cache Chunk1 Key Chunk2 Key Chunk3 Key ઃఆʹΑͬͯConsistent HashʹΑΓɺ ChunkͷKeyΛݩʹ෼ࢄͯ͠อଘͰ͖Δ

Slide 59

Slide 59 text

59 Write Ahead Log

Slide 60

Slide 60 text

60 ChunkͷFlush(࠶ܝ) Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε

Slide 61

Slide 61 text

61 IngesterͷChunkͷ࣋ͪํ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Flush͞ΕΔલʹϓϩηε͕ࢭ·Δͱ Chunk͕شൃ͢Δ

Slide 62

Slide 62 text

62 WALͷҙٛ ӬଓԽͯ͠Memoryͷشൃʹඋ͑Δ

Slide 63

Slide 63 text

63 LokiͷWALͷಛ௃ ● ϩάड৴࣌ʹɺMemoryͱWAL྆ํʹॻ͖ࠐΉ ● WALॻ͖ࠐΈ͕ࣦഊͯ͠΋ॲཧΛࣦഊͤ͞ͳ͍ ● Ingesterͷϓϩηεىಈ࣌ʹWAL͔Βͷ෮چॲཧ͕ೖΔ ● Ұఆظؒ͝ͱʹෆཁͳWAL͸ύʔδ͞ΕΔ

Slide 64

Slide 64 text

64 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Log Entry ϩάΛૹ৴

Slide 65

Slide 65 text

65 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Append SegmentϑΝΠϧʹ rawσʔλͷ··௥ه͞ΕΔ

Slide 66

Slide 66 text

66 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Create! 1ϑΝΠϧ͕େ͖͘ͳͬͯ͘Δͱ ผͷSegmentϑΝΠϧΛ࡞੒ Segment2

Slide 67

Slide 67 text

67 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Goroutine Disk Segment1 Segment2 Ұఆظؒ͝ͱʹ ෆཁͳSegmentϑΝΠϧͷ ύʔδॲཧ͕ೖΔ

Slide 68

Slide 68 text

68 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Segment2 Segment3 Create! SegmentΛҰͭਐΊΔ

Slide 69

Slide 69 text

69 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 IngesterͷະFlush ChunkΛ Checkpointͱݺ͹ΕΔ εφοϓγϣοτͱͯ͠อଘ

Slide 70

Slide 70 text

70 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment1 Segment2 Segment3 Checkpoint1 MemoryChunkߏ଄ͷ··อଘ͢Δ ͷͰblock͸ѹॖܗࣜɺ headঢ়ଶͷϩά͸ඇѹॖͱͳΔ

Slide 71

Slide 71 text

71 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment3 Checkpoint1 ࠷৽ͷCheckpointͱSegmentΛ࢒ ͯ͢͠΂ͯ࡟আ͢Δ

Slide 72

Slide 72 text

72 WALͷ࢓૊Έ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Goroutine Disk Segment3 Checkpoint1 ͦͷ࣌఺ͰFlush͞Ε͍ͯͳ͍ ͢΂ͯͷϩάؚ͕·Ε͍ͯΔ

Slide 73

Slide 73 text

73 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ

Slide 74

Slide 74 text

74 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Stream1 Stream2 Stream1 Stream2 Memory಺ʹ෮ݩ͢Δ

Slide 75

Slide 75 text

75 WALͷ࢓૊Έ Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Stream1 Stream2 Stream1 Stream2 ෮ݩ͕׬ྃ͢Δ·Ͱ͸ϓϩηεΛىಈ͠ͳ͍

Slide 76

Slide 76 text

76 Ingester্ͷσʔλͱEncodeܗࣜ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Disk Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷

Slide 77

Slide 77 text

77 2) ϩάͷಡΈࠐΈϓϩηε

Slide 78

Slide 78 text

78 Overview

Slide 79

Slide 79 text

79 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend Queriers Query Result Cache Ingesters

Slide 80

Slide 80 text

80 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ϦΫΤετΛड෇͚Δ

Slide 81

Slide 81 text

81 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ड͚औͬͨΫΤϦΛ࣌ؒͳͲͰ ෼ׂͯ͠ΩϡʔΠϯά͢Δ

Slide 82

Slide 82 text

82 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ෳ਺ͷQuerier͕Ωϡʔ͔ΒQuery Λड͚औΓϋϯυϦϯά͢Δ Queriers

Slide 83

Slide 83 text

83 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ͢΂ͯͷIngesterʹରͯ͠ɺ MemoryChunk͔ΒQueryʹMatch ͢Δ΋ͷΛཁٻ Queriers Ingesters

Slide 84

Slide 84 text

84 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend QueryʹରԠ͢ΔసஔIndexΛऔಘ͢Δ ͜ͷͱ͖CacheʹಁաతʹΞΫηε Queriers

Slide 85

Slide 85 text

85 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend Cacheʹଘࡏ͠ͳ͍৔߹͸ɺ ϩʔΧϧͷBoltDB͔ΒMatch͢ΔIndexΛऔಘ ͦͯ݁͠ՌΛCacheʹอଘ͢Δ(snappy) Queriers BoltDB

Slide 86

Slide 86 text

86 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend సஔIndex͔Βର৅ͷChunkΛׂΓग़͢ Queriers

Slide 87

Slide 87 text

87 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ChunkΛऔಘ͢Δ Cacheʹͳ͍΋ͷ͸Storage͔Βऔಘ͠ɺ Cacheʹอଘ͢Δ Queriers

Slide 88

Slide 88 text

88 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ͢΂ͯͷQuerier͔Βͷ݁ՌΛड͚औΓɺ ू໿ɺιʔτɺॏෳഉআΛ࣮ࢪ Queriers

Slide 89

Slide 89 text

89 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend ݁ՌΛQuery Result Cache΁อଘ͢Δ Queriers Query Result Cache

Slide 90

Slide 90 text

90 ಡΈࠐΈϓϩηεͷొ৔ਓ෺ Amazon S3 Chunk Cache Index Read Cache Loki Storage Query Frontend Ϩεϙϯεฦ٫ Queriers Query Result Cache

Slide 91

Slide 91 text

91 సஔIndex͔Βର৅Chunkͷબఆ

Slide 92

Slide 92 text

92 సஔIndexͷ໨త S3͔ΒChunkΛ࠷খ࿑ྗͰऔಘ͢Δ͜ͱ

Slide 93

Slide 93 text

93 సஔIndex͔Βର৅ChunkͷߜΓࠐΈ 1.LabelͷKeyͱValueͷ૊Έ߹Θ͔ͤΒStreamͷID (Series ID)Λऔಘ͢Δ 2.Series IDͱ࣌ؒൣғ͔ΒChunkͷKeyΛऔಘ͢Δ 3.ChunkͷKey͔ΒS3্ɺMemcached্ͷύεΛׂ Γग़͠ɺ௚઀ChunkΛDownload

Slide 94

Slide 94 text

94 SeriesID {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body key, valueͷ૊Έ߹Θͤͷsha256 9ac2adda49e899b312a9abb895656b1ab26c9858fd500f2ae3983d5309b39363/ ϩά SeriesID

Slide 95

Slide 95 text

95 Chunk Key {service=“keystone”, hostname=“host1”} 00:00:02 keystone log body Tenant1/a6965cd7:Chunk։࢝࣌ؒ:Chunkऴྃ࣌ؒ key, valueͷ૊Έ߹ΘͤͷHash஋ a6965cd7 ϩά FingerPrint Chunk Key

Slide 96

Slide 96 text

Hash Value Range Value Value TenantID + LabelName Hash(Label Value) + SeriesID Label Value సஔIndexͷߏ଄Πϝʔδ Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index

Slide 97

Slide 97 text

Hash Value Range Value Value TenantID + LabelName Hash(Label Value) + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index Hash + RangeͰϢχʔΫߦΛಛఆ͢Δ సஔIndexͷߏ଄Πϝʔδ

Slide 98

Slide 98 text

Hash Value Range Value Value TenantID + LabelName Hash(Label Value) + SeriesID Label Value Label Key-ValueͷHash͔ΒSeriesIDΛҾ͘Index సஔIndexͷߏ଄Πϝʔδ Range Value͸ൣғݕࡧɺιʔτʹར༻Ͱ͖Δ

Slide 99

Slide 99 text

Hash Value (TenantID + Label Name) Range Value (Hash(Label Value) + SeriesID) Value (Label Value) Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler ςʔϒϧΠϝʔδ సஔIndexͷߏ଄Πϝʔδ

Slide 100

Slide 100 text

͜ͷIndex͸ LabelͷKeyͱValueͷύλʔϯ෼͚ͩ࡞ΒΕΔ ΧʔσΟφϦςΟͷߴ͍ϥϕϧ͸ ͜ͷIndex͕େྔʹ࡞ΒΕΔ͜ͱʹͳΔ సஔIndexͷߏ଄Πϝʔδ

Slide 101

Slide 101 text

SeriesID͔ΒChunk KeyΛҾ͘Index సஔIndexͷߏ଄Πϝʔδ Hash Value Range Value Value TenantID + SeriesID Chunkͷ։࢝࣌ؒ + Chunk Key nil

Slide 102

Slide 102 text

102 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} |= “level=ERROR” LogQL ϥϕϧϚον෦ ϑΟϧλ෦

Slide 103

Slide 103 text

103 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} |= “level=ERROR” LogQL ϥϕϧϚον෦ ϑΟϧλ෦ IndexΛ࢖͏ͷ͸͜ͷ෦෼ͷධՁ

Slide 104

Slide 104 text

104 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} ෼ׂ

Slide 105

Slide 105 text

105 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} ͦΕͧΕͰϚονϯά৚݅ ͔ΒసஔIndexΛऔಘ సஔIndex సஔIndex

Slide 106

Slide 106 text

Hash Value (TenantID + Label Name) Range Value (Hash(Label Value) + SeriesID) Value Tenant1:service abc680ab:c79abadeff keystone Tenant1:host cfe960ab:bcfe12ea hostname1 Tenant1:type can860ab:c79abadeff api Tenant1:service cdc680ab:c79abadeff nova Tenant1:host bee960ab:bcfe12ea hostname21 Tenant1:type abd860ab:c79abadeff scheduler Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”} ώοτ͢Δͷ͸͜ͷϨίʔυ

Slide 107

Slide 107 text

Slide 108

Slide 108 text

108 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} Index͔ΒSeriesIDΛநग़ ྆ํʹڞ௨͢Δ΋ͷ͚ͩ࠾༻ SeriesIDs సஔIndex సஔIndex

Slide 109

Slide 109 text

109 Index͔ΒͷChunk KeyׂΓग़͠ {service=“keystone”, hostname=“host1”} {service=“keystone”} {hostname=“host1”} SeriesIDͱ࣌ؒൣғ͔Β Chunk KeyΛநग़ SeriesIDs సஔIndex Chunk Keys సஔIndex సஔIndex

Slide 110

Slide 110 text

SeriesID = c79abadeff, ൣғ=2021/10/26 21:52:00 + 5min Hash Value (TenantID + SeriesID) Range Value (Chunk։࢝࣌ؒ + Chunk Key) Value Tenant1:c79abadeff 1635252768:chunk1 nil Tenant1:c79abadeff 1635256368:chunk2 nil Tenant1:c79abadeff 1635260768:chunk3 nil Index͔ΒͷChunk KeyׂΓग़͠

Slide 111

Slide 111 text

Slide 112

Slide 112 text

112 Ϩεϙϯεͷੜ੒ͱฦ٫

Slide 113

Slide 113 text

113 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks ॳظঢ়ଶͰ͸࣮ମΛ࣋ͨͣɺ ಡΈࠐΈ໋ྩ͕͞ΕͨλΠϛϯάͰετϨʔδ(Ωϟογϡ) ʹChunkΛऔΓʹߦ͘Lazy ChunkΛੜ੒

Slide 114

Slide 114 text

114 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Ingester͔ΒͷChunkΛ͋Θͤͯɺ IteratorΛੜ੒

Slide 115

Slide 115 text

115 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Ϩεϙϯε͕نఆ݅਺ʹୡ͢Δ·Ͱɺ IteratorΛಡΈࠐΜͰ͍͘ ϩάͷFilter৚݅͸͜͜ͰධՁ͞ΕΔ |= “level=ERROR”

Slide 116

Slide 116 text

116 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response LazyChunkΛಡΈࠐΉ৔߹͸ɺ Ωϟογϡ΁ɺͳ͚Ε͹Storage΁ ChunkΛ໰͍߹ΘͤΔ Amazon S3 Chunk Cache

Slide 117

Slide 117 text

117 Ϩεϙϯεͷੜ੒ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Storage͔ΒಡΈࠐΜͩ৔߹͸ɺ औಘޙʹCacheʹอଘ͢Δ Amazon S3 Chunk Cache

Slide 118

Slide 118 text

118 Query Sharding

Slide 119

Slide 119 text

119 Queryͷ෼ׂઓུ 1. ࣌ؒ͝ͱʹ෼ׂ͢Δ ● 1࣌ؒ෼ͷϩάΛݕࡧ͢Δ৔߹ɺ15෼Ͱ෼ׂ͢ΔઃఆͳΒ4 ͭͷΫΤϦʹ෼ղ͞Ε࣮ͯߦ͞ΕΔ 2. సஔIndexΛSharding͢Δ ● ͋Β͔͡ΊసஔIndexʹShard൪߸Λ͍Ε͓͖ͯɺ QueryFrontend͕QueryΛ෼ׂ͠ɺͦΕͧΕʹShard൪߸Λ ૠೖͯ͠QuerierʹΘͨ͢

Slide 120

Slide 120 text

Hash Value Range Value Value TenantID + LabelName Shard Number + Hash(Label Value) + SeriesID Label Value సஔIndex΁ͷShard൪߸ຒΊࠐΈ Shard Number = SeriesID % shard count

Slide 121

Slide 121 text

121 సஔIndex΁ͷShard൪߸ຒΊࠐΈ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 Shard Number Stream

Slide 122

Slide 122 text

122 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”} |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 QueryΛShardͰ෼ׂ

Slide 123

Slide 123 text

123 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”} |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys औΕΔChunk͕shardͰ෼ׂ͞ΕΔ

Slide 124

Slide 124 text

124 Shard൪߸ʹΑΔQuery෼ׂ {service=“keystone”, hostname=“host1”} {service=“keystone”} {service=“keystone”,hostname=“host2”} 1 12 16 {service=“keystone”} |= “level=ERROR” LogQL Querier for shard 1 Querier for shard 12 Querier for shard 16 Chunk Keys Chunk Keys Chunk Keys |= “level=ERROR”ͷfilterॲཧΛ෼ׂॲཧͰ͖Δ

Slide 125

Slide 125 text

125 BoltDB ShipperʹΑΔIndex؅ཧ

Slide 126

Slide 126 text

126 BoltDB Shipper Ingester Shipper Disk Querier Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3

Slide 127

Slide 127 text

127 BoltDB Shipper - Ingester side Ingester Shipper Disk Querier Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 Ұఆ࣌ؒ͝ͱʹϩʔΧϧDiskʹ͋ΔIndex ΛS3΁Ξοϓϩʔυ͢Δ Ξοϓϩʔυޙʹ࡟আ͢Δ

Slide 128

Slide 128 text

128 BoltDB Shipper - Querier side Ingester Shipper Disk Querier Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 • ىಈ࣌ʹS3ʹ͋ΔIndexΛDownload • Query࣌ʹ଍Γͳ͍Index͸S3͔Β౎౓ μ΢ϯϩʔυ • Ұఆ࣌ؒ͝ͱʹ࠷ऴ࢖༻͔Β CacheTTLܦաͨ͠IndexΛ࡟আ

Slide 129

Slide 129 text

129 BoltDB Shipper Ingester Shipper Disk Querier Shipper Disk Index 1 Index 2 Index 1 Index 2 Amazon S3 Ingester, Querier͸ϩʔΧϧͰIndexΛѻ͍ɺ Shipper͕ඇಉظͰIndexΛStorageͱಉظ͢Δ

Slide 130

Slide 130 text

130 ֤ίϯϙʔωϯτͷ໾ׂ·ͱΊ

Slide 131

Slide 131 text

131 Name ໾ׂ λΠϓ σʔλͷ࣋ͪԽ ΫϥελϦϯά༗ແ Distributor όϦσʔγϣϯͱIngester΁ͷϧʔςΟϯά Stateless ༗ Ingester σʔλͷόοϑΝϦϯάͱFlush Stateful Memory: Chunks(raw + ѹॖ) Disk: WAL(raw + ѹॖ) సஔIndex(ѹॖ) ༗ Query Frontend ΫΤϦͷ෼ׂɺΩϡʔ੍ޚ Stateless ແ Querier ΫΤϦͷ࣮ߦ Stateful Disk: సஔIndexͷCache(ѹॖ) ແ Chunk Cache ChunkͷΩϟογϡ Stateful Memory: Chunks(ѹॖ) ༗(ΫϥΠΞϯταΠυ) Index Read Cache IndexͷRead༻Ωϟογϡ Stateful Memory: సஔIndex(Snappy) ༗(ΫϥΠΞϯταΠυ) Index Write Cache ಉ͡Indexͷॻ͖ࠐΈ͕ෳ਺ൃੜ͠ͳ͍Α͏ʹ ͢ΔͨΊͷ੍ޚ༻Ωϟογϡ (BoltDB ShipperͰ͸ෆཁ) Stateful Memory: Chunk Key(raw) ༗(ΫϥΠΞϯταΠυ) Query Result Cache ΫΤϦͷ݁ՌͷΩϟογϡ Stateful Memory: Query Result(raw) ༗(ΫϥΠΞϯταΠυ)

Slide 132

Slide 132 text

132 3) ো֐࣌ͷڍಈΛ஌Δ

Slide 133

Slide 133 text

133 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ

Slide 134

Slide 134 text

134 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ S3ো֐࣌ʹඋ͑Δ

Slide 135

Slide 135 text

135 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk Segment1 Checkpoint1 ͕ࣦ͜͜ഊ సஔIndex

Slide 136

Slide 136 text

136 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 Flush͸ඇಉظͳͷͰ ॻ͖ࠐΈࣗମ͸ࣦഊ͠ͳ͍

Slide 137

Slide 137 text

137 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ MemoryͱDiskʹͨ·Γଓ͚Δ

Slide 138

Slide 138 text

138 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Chunk Chunk Goroutine Goroutine Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ͕ࣦ͜͜ഊ ઌʹMemory͕͋;ΕɺOOM

Slide 139

Slide 139 text

139 WALͷ࢓૊Έ(࠶ܝ) Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Ingesterͷىಈ࣌ʹ͸ Disk͔Β SegmentͱCheckpointΛಡΈऔΔ

Slide 140

Slide 140 text

140 WALͷ࢓૊Έ(࠶ܝ) Ingester Tenant1 Memory Tenant2 Goroutine Disk Segment3 Checkpoint1 Stream1 Stream2 Stream1 Stream2 Memory಺ʹ෮ݩ͢Δ

Slide 141

Slide 141 text

141 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 WAL͸࣮࣭MemoryͷSnapshot ϩʔυ੒ޭͯ͠΋͍ۙ͏ͪʹOOM ϩʔυࣦഊͨ͠Βͦ΋ͦ΋ىಈ͠ͳ͍

Slide 142

Slide 142 text

142 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Amazon S3 Index Write Cache Chunk Cache Disk సஔΠϯσοΫε Segment1 Checkpoint1 ϘτϧωοΫ͸IngesterͷMemory

Slide 143

Slide 143 text

143 Ingester Tenant1 Memory Stream1 Stream2 Tenant2 Stream1 Stream2 Disk Segment1 Segment2 Segment3 Checkpoint1 1block sizeҎԼͷraw + ѹॖ per Chunk raw Memory Chunkͱಉ༷ Ingester্ͷσʔλͱEncodeܗࣜ(࠶ܝ)

Slide 144

Slide 144 text

144 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ ରࡦ1. ଱͍͑ͨ࣌ؒ෼ͷMemoryΛੵΉ ● ࣌ؒ͋ͨΓͷϩάྔ / ѹॖൺ * ࣌ؒ * replication factor ※ѹॖൺ͸ฐ؀ڥͰ͸gzipѹॖͰ10~18ഒͷѹॖൺ

Slide 145

Slide 145 text

145 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ 1೔ͷྲྀྔ10TBͷ؀ڥͰ1࣌ؒ଱͑Δ(ฐࣾͷ1Ϧʔδϣϯ) 1000 / 24 = 41.6 GB / hour ● Replication Factor = 1 ● Ingester * 11୆ ● Memory: 4GiB ● Disk: 8GiB(ϚʔδϯΛऔͬͯMemoryͷ2ഒ) ● Chunk Cache * 14୆ ● Memory: 3GiB

Slide 146

Slide 146 text

146 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ Replication Factor͸1Ͱ͍͍ͷ͔ʁ ● Flush͞ΕΔ·ͰʹIngesterϓϩηε͕μ΢ϯ͢Δͱͦͷؒ ͚ͩͦʹ͋ͬͨϩά͸ܽଛ͢Δ ͋Δఔ౓ׂΓ੾ΓΛ͢Δ ● WAL͕͋ΔͷͰ࠶ىಈޙʹ͙͢ʹ෮چͰ͖Δ ● ࡉ͔͍ܽଛΑΓো֐࣌ʹՔಇܧଓͰ͖ΔՄೳੑΛߴΊΔํ ޲ʹৼΔ

Slide 147

Slide 147 text

147 ॻ͖ࠐΈ࣌ͷ଱ো֐ઃܭ ରࡦ2. ো֐࣌͸Ұ࣌తʹWALΛແޮʹͯ͠ىಈ͢Δ ● WALϩʔυ͕૸Βͳ͍ͷͰɺMemory͔Β͋;ΕΔྔཷ·͍ͬͯ ͯ΋ϓϩηεىಈͰ͖ΔΑ͏ʹͳΔ ● ࠶༗ޮʹ͢Δͱ͖ʹϩά͕شൃ͠ͳ͍Α͏ɺreplication factorɺ update strategyʹ഑ྀ͢Δ

Slide 148

Slide 148 text

148 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ

Slide 149

Slide 149 text

149 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ Chunk Keys Lazy Chunks Ingester͔ΒͷChunks Iterator Response Amazon S3 Chunk Cache

Slide 150

Slide 150 text

150 ಡΈऔΓ࣌ͷ଱ো֐ઃܭ Storageো֐࣌ʹ΋ϩάΛݕࡧ͢ΔͨΊͷ৚݅ ● Ingester͕࠷௿Ұͭ͸݈ࡏͰ͋Δ͜ͱ ● ݕࡧ݁ՌΛCache͔IngesterͷσʔλͰΧόʔͰ͖Δ͜ͱ ● Cacheʹͳ͍࣌ؒൣғΛΫΤϦʹࢦఆ͠ͳ͍͜ͱ

Slide 151

Slide 151 text

151 ·ͱΊ

Slide 152

Slide 152 text

152 ·ͱΊ LokiͷίΞͰ͋Δॻ͖ࠐΈͱಡΈࠐΈϓϩηεΛৄղ ● ಈ࡞ݪཧ͕Θ͔ͬͨ͜ͱͰɺτϥϒϧγϡʔςΟϯά΍νϡ ʔχϯά͕Մೳʹ ● Ͳ͜ͰͲͷΤϯίʔσΟϯάͰσʔλΛ͔࣋ͭΛ೺Ѳ͢Δ͜ ͱͰΩϟύγςΟϓϥϯχϯά͕Մೳʹ ● ো֐࣌ͷڍಈΛ೺Ѳ͢Δ͜ͱͰద੾ͳ४උ͕ݕ౼Մೳʹ

Slide 153

Slide 153 text

153 ·ͱΊ ఻͑ΒΕͳ͔ͬͨ͜ͱ ● ϩά͔ΒͷϝτϦΫεੜ੒΍ΞϥʔςΟϯά ● ϩάͷϦςϯγϣϯ؅ཧʹ͍ͭͯ ● LokiࣗମͷϞχλϦϯάʹ͍ͭͯ ● ֤ίϯϙʔωϯτͷΩϟύγςΟ؅ཧʹ͍ͭͯ ● Out of order entry໰୊΁ͷରࡦʹ͍ͭͯ