Share
Embed
LINE ShopチームでのSREの取り組み / SRE in LINE Shop team
2020/11/7に行われたJJUG CCC 2020 Fallでのスポンサーセッションの登壇資料です。
https://ccc2020fall.java-users.jp/
LINE Developers
November 07, 2020
Transcript
75 slides
-
LINE ShopνʔϜͰͷSREͷऔΓΈ 2020/11/07 JJUG CCC 2020 Fall ( https://ccc2020fall.java-users.jp )
LINE Fukuokaגࣜձࣾ ։ൃ1ࣨ দ࡚ ֶ -
αʔϏεن • ελϯϓʹؔ͢Δࣈ *1 • ൢചதͷελϯϓ: 855ສηοτ (20203݄࣌) • 1͋ͨΓͷελϯϓૹ৴:
ฏۉ4ԯ3,300ສճ (2019݄̐࣌) •RPS(requests/sec) *2 • ීஈͷϐʔΫ: ~ 80K RPS (2020/10࣌) • ࢝ͷϐʔΫ: ~ 120K RPS (2020/01࣌) *1 https://linecorp.com/ja/pr/news/ja/2020/3127 *2 https://logmi.jp/tech/articles/322924 -
Armeria ػೳ֓ཁ •Asynchronous and reactive (like Spring WebFlux) •HTTP/2 •REST
API͚ͩͰͳ͘ɺgRPCͱThriftαϙʔτ •Client side load balancing • https://armeria.dev/docs/client-service-discovery •ϚΠΫϩαʔϏεͰඞཁͳػೳΛఏڙ • Circuit breaker, Service discovery(DNS etc),Distributed tracing(Zipkin integration), etc -
Armeria ػೳ֓ཁ •SwaggerͷΑ͏ͳdebug console •Integration • Spring Boot integration •
طଘͷJava webΞϓϦʹΈࠐΜͰ͏ࣄ͕ग़དྷΔ •etc -
Armeria ࢀߟࢿྉ •Official site: https://armeria.dev •GitHub repo: https://github.com/line/armeria •LINE DEVELOPER
DAY 2019 ʮArmeriaɿͲ͜ͰཱͭϚΠΫϩαʔϏεϑϨʔϜϫʔΫʯ • https://linedevday.linecorp.com/jp/2019/sessions/D2-2 • https://youtu.be/lii7oNzAOx0 • https://speakerdeck.com/line_devday2019/armeria-a- microservice-framework-well-suited-everywhere -
ϚΠΫϩαʔϏεʹ͓͍ͯݕ౼͕ඞཁͳࣄ •Distributed Tracing •Cascading FailureΛ͙ͨΊͷCircuit Breaker •Graceful DegradationΛߟྀͨ͠αʔϏεׂ •Service Discovery
https://employment.en-japan.com/engineerhub/entry/2018/10/09/110000 -
Graceful Degradation •ো͕ൃੜͨ͠߹ʹɺϨεϙϯεͷҰ෦ͷ࣭ΛԼʢe.g. Ωϟο γϡ͞Εͨݹ͍σʔλΛ͏ʣͤ͞Δ͜ͱͰɺຊʹେࣄͳ෦ΛकΔ •ϚΠΫϩαʔϏεͷ߹ҎԼΛҙࣝͯ͠αʔϏεΛׂ • αʔϏεʹো͕ൃੜͨ͠߹Ͱܧଓ͍ͨ͠ػೳԿ͔ʁ • Ұ෦ͷػೳϨεϙϯεͷ࣭ΛԼͤͯ͞ܧଓ͕Մೳ͔ʁ
ࢀߟࢿྉ: SRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά P281. 22.2.2 ϩʔυγΣσΟϯάͱάϨʔεϑϧσάϥϨʔγϣϯ -
Central Dogma Ϣʔεέʔε •ಈతʹઃఆ͍ͨ͠ͷΛCentral DogmaͰཧ • e.g. Service discovery, Rate
limitઃఆ, A/Bςετ, etc -
Central Dogma ࢀߟࢿྉ •Official site: https://line.github.io/centraldogma/ •GitHub repo: https://github.com/line/centraldogma/ •LINE
DEVELOPER DAY 2017 Central DogmaɿLINE ͷ GitΛϕʔεʹͨ͠ߴՄ༻ੑαʔϏεߏϨϙ δτϦ • https://www.slideshare.net/linecorp/central-dogma-lines-gitbacked- highlyavailable-service-configuration-repository • https://www.youtube.com/watch?v=BmgizIFwMq4 -
SLI • SLI (Service Level Indicator) • API Availability (ϦΫΤετޭ:
ޭ/τʔλϧϦΫΤετ) • ϨΠςϯγ • etc • SLO (Service Level Objective) • SLIΛϕʔεʹͨ͠αʔϏεͷ৴པੑͷඪ • SLO 100%ؒҧͬͨඪ • ػೳվળɺ৽ػೳՃɺϝϯςφϯε͕ߦ͑ͳ͘ͳΔ • ͍ͬͯΔϓϥοτϑΥʔϜͷSLA͕100%Ͱͳ͍߹͋Δ -
SLI • LINE ShopͰAPI availability(ޭ), API latencyΛSLIͱͯ͠༻ • ʮThe Site
Reliability Workbookʢຊޠ൛ɿαΠτϦϥΠΞϏϦ ςΟϫʔΫϒοΫʣʯʹܝࡌ͞Ε͍ͯΔࣄྫΛࢀߟʹͨ͠ • Prometheus+GrafanaͰՄࢹԽ • αʔϏεো͕ൃੜͨ࣌͠ʹϢʔβͷӨڹΛ֬ೝ͍ͯ͠Δ • SREͷϓϥΫςΟεͰɺSLOεςʔΫϗϧμʔͱ ߹ҙ͢Δඞཁ͕͋Δ͕·ͩग़དྷ͍ͯͳ͍ʢࠓޙͷ՝ʣ -
ʮSRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά - ୈᶙ෦ ࣮ફʯΑΓ αʔϏεͷ৴པੑͷ֊ • 7. ϓϩμΫτ
• 6. ։ൃ • 5. ΩϟύγςΟϓϥϯχϯά • 4. ςετٴͼϦϦʔεखॱ • 3. ϙετϞʔςϜ/ࠜຊݪҼੳ • 2. ΠϯγσϯτରԠ • 1. ϞχλϦϯά -
ʮSRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά - ୈᶙ෦ ࣮ફʯΑΓ αʔϏεͷ৴པੑͷ֊ • 7. ϓϩμΫτ
• 6. ։ൃ • 5. ΩϟύγςΟϓϥϯχϯά • 4. ςετٴͼϦϦʔεखॱ • 3. ϙετϞʔςϜ/ࠜຊݪҼੳ • 2. ΠϯγσϯτରԠ • 1. ϞχλϦϯά -
ʮSRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά - ୈᶙ෦ ࣮ફʯΑΓ αʔϏεͷ৴པੑͷ֊ • 7. ϓϩμΫτ
• 6. ։ൃ • 5. ΩϟύγςΟϓϥϯχϯά • 4. ςετٴͼϦϦʔεखॱ • 3. ϙετϞʔςϜ/ࠜຊݪҼੳ • 2. ΠϯγσϯτରԠ • 1. ϞχλϦϯά -
ϞχλϦϯά - Alerting • ErrorʢϢʔβʹతͳӨڹ͋Γʣ • LatencyͷѱԽ • Error response/secͷ૿Ճ
• etc • WarnʢͷݪҼͱͳΔͷ or αʔϏεӨڹ͕͍ʣ • CPU usage • JVM GC • ΞϓϦέʔγϣϯαʔό͕མͪͨʢͳΒαʔϏεӨڹແ͍ʣ • etc -
ϞχλϦϯά - ऩू͍ͯ͠ΔϝτϦΫε • API͝ͱͷϝτϦΫε • Server/Client latency (50th, 90th,
99th percentile, etc) • Requests/sec • Error responses/sec • ϩάͷྔ (Warn, Error) • JVM (GC, Heap, etc) • DB client metrics (HikariCP, etc) • Server load (CPU, Memory, Network Traffic, etc) • etc… -
Armeria͕export͢ΔϝτϦΫεͷྫʢҰ෦ʣ • Server/Client latency (50th, 90th, 99th percentile, etc) •
Requests/sec • Error response/sec • Circuit breaker(CLOSED, OPEN, HALF_OPEN, etc) -
Armeria͕export͢ΔϝτϦΫεͷྫʢҰ෦ʣ • Request/Response size • ݺͼग़͠ଆͷͰRequest size͕૿͑ͯαʔόͷෛՙ্͕͕ ΔՄೳੑΛϞχλϦϯά • αʔόଆͷͰෆਖ਼ͳʢۃʹখ͞ͳʣϨεϙϯεΛฦͯ͠
͠·͏ՄೳੑΛϞχλϦϯά • Armeria client͕DNS໊લղܾʹ͔͔ͬͨ࣌ؒ • DNS͕ݪҼͰ໊લղܾʹ͕͔͔࣌ؒΓϨΠςϯγ͕ѱԽͯ͠͠ ·͏ՄೳੑΛϞχλϦϯά -
ΞϓϦέʔγϣϯݻ༗ͷϝτϦΫεͷྫ • όʔδϣϯʢ͍ͭɺͲͷόʔδϣϯ͕σϓϩΠ͞Εͨͷ͔ʁʣ • ΞϓϦέʔγϣϯ • ϑϨʔϜϫʔΫ • Armeria •
Spring Boot • JVM • etc… -
ϞχλϦϯά - Batch job Metrics: shop_batch_successful_time_seconds{job=“foo”, period="10min"} 1601313019 ※”1601313019”ͷ෦job͕ਖ਼ৗʹྃͨ࣌͠ͷUNIX time
Alert rule: time() - shop_batch_successful_time_seconds{period="10min"} > 60 * 10 * 3 ※”3”Ұ࣌తͳΤϥʔͰΞϥʔτΛ্͛ͳ͍ͨΊͷόοϑΝɻ͓ΈͰɻ ※͜ͷྫͰɺperiod=“10min”ϥϕϧΛ͚ͨbatch job͕લճऴྃҎ߱ɺ30Ҏʹਖ਼ৗऴ ྃ͠ͳ͚ΕΞϥʔτ্͕͕Δ https://www.robustperception.io/monitoring-batch-jobs-in-python -
ʮSRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά - ୈᶙ෦ ࣮ફʯΑΓ αʔϏεͷ৴པੑͷ֊ • 7. ϓϩμΫτ
• 6. ։ൃ • 5. ΩϟύγςΟϓϥϯχϯά • 4. ςετٴͼϦϦʔεखॱ • 3. ϙετϞʔςϜ/ࠜຊݪҼੳ • 2. ΠϯγσϯτରԠ • 1. ϞχλϦϯά -
on-call • on-call୲Λຖि2ਓͰ࣋ͪճΓ • νʔϜͰ࡞ͨ͠ΨΠυΛݩʹΞϥʔτͷରԠΛߦ͏ • ൃੜͨ͠ΞϥʔτΛνΣοΫ͠ɺϝϯόʔʹΤεΧϨʔγϣϯ • on-call୲Ҏ֎ͷϝϯόʔΞϥʔτରԠʹࢀՃ •
֤छμϯϓɾϩάͳͲΛऔಘ • JVM Thread dump • JVM Heap dump • JFRϩά • etc -
ʮSRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά - ୈᶙ෦ ࣮ફʯΑΓ αʔϏεͷ৴པੑͷ֊ • 7. ϓϩμΫτ
• 6. ։ൃ • 5. ΩϟύγςΟϓϥϯχϯά • 4. ςετٴͼϦϦʔεखॱ • 3. ϙετϞʔςϜ/ࠜຊݪҼੳ • 2. ΠϯγσϯτରԠ • 1. ϞχλϦϯά -
ϙετϞʔςϜ • ϙετϞʔςϜʹ·ͱΊΔ߲ • Өڹൣғ • োͷݪҼ • ঢ়گͷ࣌ܥྻ·ͱΊ •
࠶ൃࢭࡦͷݕ౼ • োݕʹ͕ͳ͔͔ͬͨʁͲ͏վળ͢Δ͔ʁ • োͷϋϯυϦϯάʹ͕ͳ͔͔ͬͨʁͲ͏վળ͢Δ͔ʁ -
ʮSRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά - ୈᶙ෦ ࣮ફʯΑΓ αʔϏεͷ৴པੑͷ֊ • 7. ϓϩμΫτ
• 6. ։ൃ • 5. ΩϟύγςΟϓϥϯχϯά • 4. ςετٴͼϦϦʔεखॱ • 3. ϙετϞʔςϜ/ࠜຊݪҼੳ • 2. ΠϯγσϯτରԠ • 1. ϞχλϦϯά -
ΩϟύγςΟϓϥϯχϯά • ݩ୴ͷ0:00ޙ࠷τϥϑΟοΫ͕૿͑ΔΠϕϯτͷ̍ͭ • ຖࣄલ४උΛߦ͍ͬͯΔ • ڈͷݩ୴ɺͦͷଞͷΠϕϯτͷϝτϦΫεΛݩʹαʔόͷ εέʔϧΞοϓεέʔϧΞτΛߦ͏ • LINE
Developer MeetupͰൃදͨ࣌͠ͷॻ͖ى͕͋͜͠Γ·͢ • https://logmi.jp/tech/articles/322924 -
ಥൃతͳաෛՙͷରॲ • ಥൃతʹൃੜͨ͠աෛՙ͕ݪҼͰαʔϏεো͕ൃੜ͢ΔՄೳੑ • ߟ͑ΒΕΔ͍͔ͭ͘ͷཁҼ • TVͰऔΓ্͛ΒΕͯಥൃతͳεύΠΫ͕ൃੜ • ෆ۩߹ΛؚΉόʔδϣϯͷσϓϩΠʹΑΔϦΫΤετ૿Ճ •
etc • Ͳ͏ରॲ͢Δ͔ʁ • Rate limit ʢҰఆͷϦΫΤετΛrejectͯ͠աෛՙঢ়ଶʹͳΔࣄΛ͙ʣ • αʔόͷεέʔϧΞτ -
LINE ShopͰͷRate limit • Rate limitͷઃఆCentral DogmaͰཧ • Rate limitॲཧArmeriaͷThrottlingServiceΛͬͯΞϓϦʹ࣮
https://armeria.dev/docs/advanced-production-checklist/ -
ͳͥLINE ShopͰk8sΛ͍͍ͨͷ͔ʁ • ΄ͱΜͲͷϚΠΫϩαʔϏεVerdaʢPrivate CloudʣͷVMͱ ཧαʔόΛ༻த • AnsibleͰϓϩϏδϣχϯά • εέʔϧΞτʹ͕͔͔࣌ؒΔ
• ಥൃతͳϦΫΤετ૿Ճ࣌ʹૉૣ͘εέʔϧΞτ͍ͨ͠ • VerdaͰk8sͷαʔϏεఏڙ͞Ε͍ͯΔ -
ݕ౼ɾ४උͨ͠ࣄʢϞχλϦϯάʣ • PrometheusαʔόΛࣗୡͰӡ༻͢ΔࣄΛආ͚͍ͨ • LINE ShopαʔϏεશମͰऩू͞Ε͍ͯΔϝτϦΫεͷ૯ (scrape_samples_scraped)6,000,000Ҏ্ • PrometheusαʔόΛ҆ఆՔಇͤ͞ΔͨΊʹPrometheusͷ ݟɾϊϋ͕ඞཁ
• ࠓͬͯΔPrometheus/Alertmanager/alert rule/Grafanaμο γϡϘʔυΛk8sڥͰͦͷ··͍͍ͨ -
·ͱΊ • LINE ShopαʔϏεΛ͝հ͠·ͨ͠ • LINE ShopαʔϏεΞʔΩςΫνϟΛ͝հ͠·ͨ͠ • ArmeriaͱCentral Dogma
• ϚΠΫϩαʔϏεʹ͓͍ͯݕ౼͕ඞཁͳࣄ • Distributed Tracing • Cascading FailureΛ͙ͨΊͷCircuit Breaker • Graceful DegradationΛߟྀͨ͠αʔϏεׂ • Service Discovery -
·ͱΊ • LINE ShopνʔϜͰͷSREͷऔΓΈΛ͝հ͠·ͨ͠ • αʔϏεͷ৴པੑͷ֊ • ϞχλϦϯά • ΠϯγσϯτରԠ
• ϙετϞʔςϜ • ΩϟύγςΟϓϥϯχϯά -
LINE DEVELOPER DAY 2020ͷ͝Ҋ • ެࣜαΠτ: https://linedevday.linecorp.com/2020/ja • ఔ: 2020/11/25~27ʢΦϯϥΠϯ։࠵ʣ
• ߹ܭ150Ҏ্ͷηογϣϯ • શηογϣϯӳ௨༁ରԠ -
We are hiring • LINE Fukuokaגࣜձࣾ • αʔόʔαΠυΤϯδχΞ https://linefukuoka.co.jp/ja/career/list/engineer/ development_engineer_server-side
• LINEגࣜձࣾ • γχΞαʔόʔαΠυΤϯδχΞ/ίϯςϯπൢചϓϥοτϑΥʔϜ https://linecorp.com/ja/career/position/665 • Site Reliability Engineer/ίϯςϯπൢചϓϥοτϑΥʔϜ https://linecorp.com/ja/career/position/1535