LINE ShopチームでのSREの取り組み / SRE in LINE Shop team

by LINE Developers

Slide 1

Slide 1 text

LINE ShopνʔϜͰͷSREͷऔΓ૊Έ 2020/11/07 JJUG CCC 2020 Fall ( https://ccc2020fall.java-users.jp ) LINE Fukuokaגࣜձࣾ ։ൃ1ࣨ দ࡚ ֶ

Slide 2

Slide 2 text

ࣗݾ঺հ @matsumana LINE Fukuokaגࣜձࣾ ։ൃ1ࣨ SRE/Server Side Engineer https://github.com/matsumana Manabu Matsuzaki

Slide 3

Slide 3 text

• LINE ShopαʔϏε঺հ • LINE ShopαʔϏεΞʔΩςΫνϟ঺հ • LINE ShopνʔϜͰͷSREͷऔΓ૊Έ Agenda

Slide 4

Slide 4 text

LINE ShopαʔϏε঺հ

Slide 5

Slide 5 text

LINE Shopͱ͸ʁ •LINEͷίϯςϯπൢചϓϥοτϑΥʔϜʢLINEͷελϯϓɾ ֆจࣈɾண͔ͤ͑ػೳʣͷࣾ಺Ͱͷ௨শ • LINEΞϓϦ಺ͷελϯϓγϣοϓɺண͔ͤ͑γϣοϓ • WebͷLINE STORE (https://store.line.me/)

Slide 6

Slide 6 text

ΧελϜελϯϓ • ελϯϓͷςΩετͷҰ෦ΛมߋՄೳͳελϯϓ • https://linecorp.com/ja/pr/news/ja/2019/2664

Slide 7

Slide 7 text

ϝοηʔδελϯϓ • ΧελϜελϯϓΑΓ΋௕จΛೖྗՄೳͳελϯϓ • https://linecorp.com/ja/pr/news/ja/2020/3127

Slide 8

Slide 8 text

LINEελϯϓ ϓϨϛΞϜ • ΫϦΤΠλʔζελϯϓͷαϒεΫϦϓγϣϯ • https://store.line.me/stickers-premium/landing/ja • http://creator-mag.line.me/ja/archives/1075007192.html

Slide 9

Slide 9 text

αʔϏεن໛ • ελϯϓʹؔ͢Δ਺ࣈ *1 • ൢചதͷελϯϓ਺: 855ສηοτ (2020೥3݄࣌఺) • 1೔͋ͨΓͷελϯϓૹ৴਺: ฏۉ4ԯ3,300ສճ (2019೥݄̐࣌఺) •RPS(requests/sec) *2 • ීஈͷϐʔΫ: ~ 80K RPS (2020/10࣌఺) • ೥࢝ͷϐʔΫ: ~ 120K RPS (2020/01࣌఺) *1 https://linecorp.com/ja/pr/news/ja/2020/3127 *2 https://logmi.jp/tech/articles/322924

Slide 10

Slide 10 text

LINEελϯϓͷLINEެࣜΞΧ΢ϯτͷ ༑ͩͪ਺ͷਪҠ • 2018/12: 39,000,000 • 2019/12: 55,000,000 • 2020/10: 63,000,000

Slide 11

Slide 11 text

LINE ShopαʔϏεΞʔΩςΫνϟ঺հ

Slide 12

Slide 12 text

LINEʹ͓͚ΔϚΠΫϩαʔϏε •LINE Shop͸ϚΠΫϩαʔϏεͰߏங͞Ε͍ͯΔ •LINEϝοηʔδϯάϓϥοτϑΥʔϜશମ͔ΒݟΔͱɺ LINE Shopࣗମ΋̍ͭͷϚΠΫϩαʔϏε *1 *1: LINEͷϝοηʔδϯάϓϥοτϑΥʔϜʹ͓͚ΔϚΠΫϩαʔϏεԽ΁ͷ௕͍ಓͷΓ https://linedevday.linecorp.com/jp/2019/sessions/D1-6

Slide 13

Slide 13 text

LINE Shopʹ͓͚ΔϚΠΫϩαʔϏεʢҰ෦ʣ

Slide 14

Slide 14 text

ϑϨʔϜϫʔΫɾϞχλϦϯά

Slide 15

Slide 15 text

Armeria ػೳ֓ཁ •Asynchronous and reactive (like Spring WebFlux) •HTTP/2 •REST API͚ͩͰ͸ͳ͘ɺgRPCͱThrift΋αϙʔτ •Client side load balancing • https://armeria.dev/docs/client-service-discovery •ϚΠΫϩαʔϏεͰඞཁͳػೳΛఏڙ • Circuit breaker, Service discovery(DNS etc),Distributed tracing(Zipkin integration), etc

Slide 16

Slide 16 text

Armeria ػೳ֓ཁ •SwaggerͷΑ͏ͳdebug console •Integration • Spring Boot integration • طଘͷJava webΞϓϦʹ૊ΈࠐΜͰ࢖͏ࣄ͕ग़དྷΔ •etc

Slide 17

Slide 17 text

Armeria ࢀߟࢿྉ •Official site: https://armeria.dev •GitHub repo: https://github.com/line/armeria •LINE DEVELOPER DAY 2019 ʮArmeriaɿͲ͜Ͱ΋໾ཱͭϚΠΫϩαʔϏεϑϨʔϜϫʔΫʯ • https://linedevday.linecorp.com/jp/2019/sessions/D2-2 • https://youtu.be/lii7oNzAOx0 • https://speakerdeck.com/line_devday2019/armeria-a- microservice-framework-well-suited-everywhere

Slide 18

Slide 18 text

Armeria ࢀߟࢿྉ •JSUGษڧձͰʮSpring BootϢʔβͷͨΊͷArmeriaೖ໳ʯͱ ͍͏λΠτϧͰLT͠·ͨ͠ • https://matsumana.info/blog/2020/07/30/introduce-to-armeria- for-spring-users/

Slide 19

Slide 19 text

LINE ShopͷϚΠΫϩαʔϏεͱ։ൃνʔϜ •౦ژͱ෱Ԭ߹ΘͤͯνʔϜϝϯόʔ͸໿25ਓʢαʔόαΠυ+SREʣ •ϓϩδΣΫτ୯ҐͰνʔϜ͕࡞ΒΕɺෳ਺ͷϚΠΫϩαʔϏεΛඞཁʹ Ԡͯ͡ػೳ௥Ճɾमਖ਼͍ͯ͠Δ

Slide 20

Slide 20 text

ϚΠΫϩαʔϏεʹ͓͍ͯݕ౼͕ඞཁͳࣄ •Distributed Tracing •Cascading FailureΛ๷͙ͨΊͷCircuit Breaker •Graceful DegradationΛߟྀͨ͠αʔϏε෼ׂ •Service Discovery https://employment.en-japan.com/engineerhub/entry/2018/10/09/110000

Slide 21

Slide 21 text

Distributed Tracing • APIͷݺͼग़͠ͱɺͦΕʹ͔͔ͬͨ࣌ؒΛՄࢹԽ͢Δ • ϨΠςϯγʹ໰୊͕͋Δ৔߹ͷϘτϧωοΫௐࠪ • LINE ShopͰ͸ɺZipkinΛ࢖༻

Slide 22

Slide 22 text

Circuit Breaker https://armeria.dev/docs/client-circuit-breaker • ݺͼग़͠ઌʹো֐͕ൃੜͨ͠৔߹ɺͦΕ͕ղফ͞ΕΔ·Ͱ௨৴ ΛߦΘͳ͍Α͏ʹ͢Δ

Slide 23

Slide 23 text

Circuit Breaker

Slide 24

Slide 24 text

Circuit Breaker ※Cascading Failure͕ൃੜ

Slide 25

Slide 25 text

Circuit Breaker

Slide 26

Slide 26 text

Circuit Breaker (ArmeriaͷFailFastException)

Slide 27

Slide 27 text

Graceful Degradation •ো֐͕ൃੜͨ͠৔߹ʹɺϨεϙϯεͷҰ෦ͷ඼࣭Λ௿Լʢe.g. Ωϟο γϡ͞Εͨݹ͍σʔλΛ࢖͏ʣͤ͞Δ͜ͱͰɺຊ౰ʹେࣄͳ෦෼ΛकΔ •ϚΠΫϩαʔϏεͷ৔߹͸ҎԼΛҙࣝͯ͠αʔϏεΛ෼ׂ • αʔϏεʹো֐͕ൃੜͨ͠৔߹Ͱ΋ܧଓ͍ͨ͠ػೳ͸Կ͔ʁ • Ұ෦ͷػೳ͸Ϩεϙϯεͷ࣭Λ௿Լͤͯ͞ܧଓ͕Մೳ͔ʁ ࢀߟࢿྉ: SRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά P281. 22.2.2 ϩʔυγΣσΟϯάͱάϨʔεϑϧσάϥϨʔγϣϯ

Slide 28

Slide 28 text

Service DiscoveryʢLINE Shopͷ৔߹ʣ

Slide 29

Slide 29 text

Central Dogma ػೳ֓ཁ •ઃఆϦϙδτϦαʔϏε •watch͓͚ͯ͠͹มߋ௨஌Λड͚औΕΔ •όοΫΤϯυʹGitΛ࢖༻ •ΫϥΠΞϯτϥΠϒϥϦ (Java, Go) •etc

Slide 30

Slide 30 text

Central Dogma Ϣʔεέʔε •ಈతʹઃఆ͍ͨ͠΋ͷΛCentral DogmaͰ؅ཧ • e.g. Service discovery, Rate limitઃఆ, A/Bςετ, etc

Slide 31

Slide 31 text

Central Dogma ࢀߟࢿྉ •Official site: https://line.github.io/centraldogma/ •GitHub repo: https://github.com/line/centraldogma/ •LINE DEVELOPER DAY 2017 Central DogmaɿLINE ͷ GitΛϕʔεʹͨ͠ߴՄ༻ੑαʔϏεߏ੒Ϩϙ δτϦ • https://www.slideshare.net/linecorp/central-dogma-lines-gitbacked- highlyavailable-service-configuration-repository • https://www.youtube.com/watch?v=BmgizIFwMq4

Slide 32

Slide 32 text

LINE ShopνʔϜͰͷSREͷऔΓ૊Έ

Slide 33

Slide 33 text

SLI

Slide 34

Slide 34 text

SLI • SLI (Service Level Indicator) • API Availability (ϦΫΤετ੒ޭ཰: ੒ޭ਺/τʔλϧϦΫΤετ਺) • ϨΠςϯγ • etc • SLO (Service Level Objective) • SLIΛϕʔεʹͨ͠αʔϏεͷ৴པੑͷ໨ඪ • SLO 100%͸ؒҧͬͨ໨ඪ • ػೳվળɺ৽ػೳ௥Ճɺϝϯςφϯε͕ߦ͑ͳ͘ͳΔ • ࢖͍ͬͯΔϓϥοτϑΥʔϜͷSLA͕100%Ͱ͸ͳ͍৔߹΋͋Δ

Slide 35

Slide 35 text

SLI • LINE ShopͰ͸API availability(੒ޭ཰), API latencyΛSLIͱͯ͠࢖༻ • ʮThe Site Reliability Workbookʢ೔ຊޠ൛ɿαΠτϦϥΠΞϏϦ ςΟϫʔΫϒοΫʣʯʹܝࡌ͞Ε͍ͯΔࣄྫΛࢀߟʹͨ͠ • Prometheus+GrafanaͰՄࢹԽ • αʔϏεো֐͕ൃੜͨ࣌͠ʹϢʔβ΁ͷӨڹΛ֬ೝ͍ͯ͠Δ • SREͷϓϥΫςΟεͰ͸ɺSLO͸εςʔΫϗϧμʔͱ ߹ҙ͢Δඞཁ͕͋Δ͕·ͩग़དྷ͍ͯͳ͍ʢࠓޙͷ՝୊ʣ

Slide 36

Slide 36 text

SLI Dashboard • ݱࡏͷঢ়ଶΛදࣔ͢ΔμογϡϘʔυ • ͖͍͠஋ͱഎܠ৭Λઃఆ͍ͯͯ͠ɺ ϝτϦΫεʹΑͬͯഎܠ৭͕มΘΔ

Slide 37

Slide 37 text

SLI Dashboard

Slide 38

Slide 38 text

ʮαʔϏεͷ৴པੑͷ֊૚ʯʹ͍ͭͯ

Slide 39

Slide 39 text

ʮSRE αΠτϦϥΠΞϏϦςΟ ΤϯδχΞϦϯά - ୈᶙ෦ ࣮ફʯΑΓ αʔϏεͷ৴པੑͷ֊૚ • 7. ϓϩμΫτ • 6. ։ൃ • 5. ΩϟύγςΟϓϥϯχϯά • 4. ςετٴͼϦϦʔεखॱ • 3. ϙετϞʔςϜ/ࠜຊݪҼ෼ੳ • 2. ΠϯγσϯτରԠ • 1. ϞχλϦϯά

Slide 40

Slide 40 text

Slide 41

Slide 41 text

Slide 42

Slide 42 text

ϞχλϦϯά - Alerting • Ϣʔβʹ௚઀తͳӨڹ͕͋ΔAlertͳͷ͔Ͳ͏͔ʹج͍ͮͯ AlertϨϕϧͱ௨஌͢ΔChannelΛ෼͚͍ͯΔ

Slide 43

Slide 43 text

ϞχλϦϯά - Alerting • ErrorʢϢʔβʹ௚઀తͳӨڹ͋Γʣ • LatencyͷѱԽ • Error response/secͷ૿Ճ • etc • Warnʢ໰୊ͷݪҼͱͳΔ΋ͷ or αʔϏεӨڹ͕௿͍ʣ • CPU usage • JVM GC • ΞϓϦέʔγϣϯαʔό͕མͪͨʢ਺୆ͳΒαʔϏεӨڹ͸ແ͍ʣ • etc

Slide 44

Slide 44 text

ϞχλϦϯά - ऩू͍ͯ͠ΔϝτϦΫε • API͝ͱͷϝτϦΫε • Server/Client latency (50th, 90th, 99th percentile, etc) • Requests/sec • Error responses/sec • ϩάͷྔ (Warn, Error) • JVM (GC, Heap, etc) • DB client metrics (HikariCP, etc) • Server load (CPU, Memory, Network Traffic, etc) • etc…

Slide 45

Slide 45 text

Armeria͕export͢ΔϝτϦΫεͷྫʢҰ෦ʣ • Server/Client latency (50th, 90th, 99th percentile, etc) • Requests/sec • Error response/sec • Circuit breaker(CLOSED, OPEN, HALF_OPEN, etc)

Slide 46

Slide 46 text

Armeria͕export͢ΔϝτϦΫεͷྫʢҰ෦ʣ • Request/Response size • ݺͼग़͠ଆͷ໰୊ͰRequest size͕૿͑ͯαʔόͷෛՙ্͕͕ ΔՄೳੑΛϞχλϦϯά • αʔόଆͷ໰୊Ͱෆਖ਼ͳʢۃ୺ʹখ͞ͳʣϨεϙϯεΛฦͯ͠ ͠·͏ՄೳੑΛϞχλϦϯά • Armeria client͕DNS໊લղܾʹ͔͔ͬͨ࣌ؒ • DNS͕ݪҼͰ໊લղܾʹ͕͔͔࣌ؒΓϨΠςϯγ͕ѱԽͯ͠͠ ·͏ՄೳੑΛϞχλϦϯά

Slide 47

Slide 47 text

ΞϓϦέʔγϣϯݻ༗ͷϝτϦΫεͷྫ • όʔδϣϯʢ͍ͭɺͲͷόʔδϣϯ͕σϓϩΠ͞Εͨͷ͔ʁʣ • ΞϓϦέʔγϣϯ • ϑϨʔϜϫʔΫ • Armeria • Spring Boot • JVM • etc…

Slide 48

Slide 48 text

ΞϓϦέʔγϣϯݻ༗ͷϝτϦΫεͷྫ https://github.com/line/armeria/blob/armeria-1.1.0/core/src/main/java/com/linecorp/armeria/server/Server.java#L375

Slide 49

Slide 49 text

ΞϓϦέʔγϣϯݻ༗ͷϝτϦΫεͷྫ Metrics: armeria_build_info{version=“1.1.0", …} 1.0 PromQL: sum(armeria_build_info{project=~”$project”, …}) by (project, version)

Slide 50

Slide 50 text

ϞχλϦϯά - Batch job Metrics: shop_batch_successful_time_seconds{job=“foo”, period="10min"} 1601313019 ※”1601313019”ͷ෦෼͸job͕ਖ਼ৗʹ׬ྃͨ࣌͠఺ͷUNIX time Alert rule: time() - shop_batch_successful_time_seconds{period="10min"} > 60 * 10 * 3 ※”3”͸Ұ࣌తͳΤϥʔͰΞϥʔτΛ্͛ͳ͍ͨΊͷόοϑΝɻ͓޷ΈͰɻ ※͜ͷྫͰ͸ɺperiod=“10min”ϥϕϧΛ෇͚ͨbatch job͕લճऴྃҎ߱ɺ30෼Ҏ಺ʹਖ਼ৗऴ ྃ͠ͳ͚Ε͹Ξϥʔτ্͕͕Δ https://www.robustperception.io/monitoring-batch-jobs-in-python

Slide 51

Slide 51 text

Slide 52

Slide 52 text

on-call • on-call୲౰Λຖि2ਓͰ࣋ͪճΓ • νʔϜͰ࡞੒ͨ͠ΨΠυΛݩʹΞϥʔτͷରԠΛߦ͏ • ൃੜͨ͠ΞϥʔτΛνΣοΫ͠ɺϝϯόʔʹΤεΧϨʔγϣϯ • on-call୲౰Ҏ֎ͷϝϯόʔ΋ΞϥʔτରԠʹࢀՃ • ֤छμϯϓɾϩάͳͲΛऔಘ • JVM Thread dump • JVM Heap dump • JFRϩά • etc

Slide 53

Slide 53 text

on-call • αʔϏεӨڹ͕͋Δ৔߹͸ؔ܎͢ΔChannelʹΤεΧϨʔγϣϯ • ൃੜͨ͠issueͷνέοτొ࿥ • αʔϏεো֐͕ൃੜͨ͠৔߹͸ϨϙʔτΛ࡞੒ • ͍ΘΏΔSREϓϥΫςΟεʹ͓͚ΔϙετϞʔςϜ • etc

Slide 54

Slide 54 text

Slide 55

Slide 55 text

ϙετϞʔςϜ • ϙετϞʔςϜͱ͸ɺαʔϏεো֐͕ൃੜͨ͠৔߹ʹॻ͘ Ϩϙʔτ΍ɺͦͷऔΓ૊Έͷࣄ • LINEͰ͸Ҏલ͔ΒϙετϞʔςϜͷจԽ͕͋Δ • LINE Shopͷ৔߹ɺϙετϞʔςϜΛॻ͍ͯؔ܎νʔϜͱ ϛʔςΟϯάΛ։࠵͍ͯ͠Δ

Slide 56

Slide 56 text

ϙετϞʔςϜ • ϙετϞʔςϜʹ·ͱΊΔ߲໨ • Өڹൣғ • ো֐ͷݪҼ • ঢ়گͷ࣌ܥྻ·ͱΊ • ࠶ൃ๷ࢭࡦͷݕ౼ • ো֐ݕ஌ʹ໰୊͕ͳ͔͔ͬͨʁͲ͏վળ͢Δ͔ʁ • ো֐ͷϋϯυϦϯάʹ໰୊͕ͳ͔͔ͬͨʁͲ͏վળ͢Δ͔ʁ

Slide 57

Slide 57 text

Slide 58

Slide 58 text

ΩϟύγςΟϓϥϯχϯά • ݩ୴ͷ0:00௚ޙ͸࠷΋τϥϑΟοΫ͕૿͑ΔΠϕϯτͷ̍ͭ • ຖ೥ࣄલ४උΛߦ͍ͬͯΔ • ڈ೥ͷݩ୴΍ɺͦͷଞͷΠϕϯτͷϝτϦΫεΛݩʹαʔόͷ εέʔϧΞοϓ΍εέʔϧΞ΢τΛߦ͏ • LINE Developer MeetupͰൃදͨ࣌͠ͷॻ͖ى͕͋͜͠Γ·͢ • https://logmi.jp/tech/articles/322924

Slide 59

Slide 59 text

ΩϟύγςΟϓϥϯχϯά • ػೳ௥Ճ΍ɺLINEελϯϓͷLINEެࣜΞΧ΢ϯτͷ༑ͩͪ਺ͷ ૿ՃʹΑͬͯɺීஈͷϦΫΤετ΋૿͑ଓ͚͍ͯΔ • ݩ୴Ҏ֎ͷλΠϛϯάͰ΋ඞཁʹԠͯ͡ਵ࣌εέʔϧΞ΢τ

Slide 60

Slide 60 text

ͦͷଞͷτϐοΫ

Slide 61

Slide 61 text

ͦͷଞͷτϐοΫ • ಥൃతͳաෛՙ΁ͷରॲ • k8sΛ࢖͏࣌ʹݕ౼ɾ४උͨ͠ࣄ

Slide 62

Slide 62 text

ಥൃతͳաෛՙ΁ͷରॲ • ಥൃతʹൃੜͨ͠աෛՙ͕ݪҼͰαʔϏεো֐͕ൃੜ͢ΔՄೳੑ • ߟ͑ΒΕΔ͍͔ͭ͘ͷཁҼ • TVͰऔΓ্͛ΒΕͯಥൃతͳεύΠΫ͕ൃੜ • ෆ۩߹ΛؚΉόʔδϣϯͷσϓϩΠʹΑΔϦΫΤετ૿Ճ • etc • Ͳ͏ରॲ͢Δ͔ʁ • Rate limit ʢҰఆͷϦΫΤετΛrejectͯ͠աෛՙঢ়ଶʹͳΔࣄΛ๷͙ʣ • αʔόͷεέʔϧΞ΢τ

Slide 63

Slide 63 text

LINE ShopͰͷRate limit • Rate limitͷઃఆ஋͸Central DogmaͰ؅ཧ • Rate limitॲཧ͸ArmeriaͷThrottlingServiceΛ࢖ͬͯΞϓϦʹ࣮૷ https://armeria.dev/docs/advanced-production-checklist/

Slide 64

Slide 64 text

k8sΛ࢖͏࣌ʹݕ౼ɾ४උͨ͠ࣄ

Slide 65

Slide 65 text

ͳͥLINE ShopͰk8sΛ࢖͍͍ͨͷ͔ʁ • ΄ͱΜͲͷϚΠΫϩαʔϏε͸VerdaʢPrivate CloudʣͷVMͱ ෺ཧαʔόΛ࢖༻த • AnsibleͰϓϩϏδϣχϯά • εέʔϧΞ΢τʹ͕͔͔࣌ؒΔ • ಥൃతͳϦΫΤετ૿Ճ࣌ʹૉૣ͘εέʔϧΞ΢τ͍ͨ͠ • VerdaͰ͸k8sͷαʔϏε΋ఏڙ͞Ε͍ͯΔ

Slide 66

Slide 66 text

ݕ౼ɾ४උͨ͠ࣄʢϞχλϦϯάʣ • k8s؀ڥʹ͓͚ΔPrometheusϞχλϦϯάͷҰൠతͳύλʔϯ → Ϋϥελ಺ʹPrometheusΛσϓϩΠ͢Δ

Slide 67

Slide 67 text

ݕ౼ɾ४උͨ͠ࣄʢϞχλϦϯάʣ • PrometheusαʔόΛࣗ෼ୡͰӡ༻͢ΔࣄΛආ͚͍ͨ • LINE ShopαʔϏεશମͰऩू͞Ε͍ͯΔϝτϦΫεͷ૯਺ (scrape_samples_scraped)͸6,000,000Ҏ্ • PrometheusαʔόΛ҆ఆՔಇͤ͞ΔͨΊʹ͸Prometheusͷ ஌ݟɾϊ΢ϋ΢͕ඞཁ • ࠓ࢖ͬͯΔPrometheus/Alertmanager/alert rule/Grafanaμο γϡϘʔυΛk8s؀ڥͰ΋ͦͷ··࢖͍͍ͨ

Slide 68

Slide 68 text

ݕ౼ɾ४උͨ͠ࣄʢϞχλϦϯάʣ • k8s API serverܦ༝ͰΫϥελ֎ͷPrometheusαʔό͔ΒϝτϦ ΫεΛऩू͢Δࣄ΋Մೳ

Slide 69

Slide 69 text

ݕ౼ɾ४උͨ͠ࣄʢϞχλϦϯάʣ • σϓϩΠ͢ΔPodͷϝτϦΫεΛk8s APIαʔόܦ༝Ͱऩू͢Δͷ ͸ආ͚͍ͨ • Pod਺͕૿͑ͨ৔߹Ͱ΋ɺk8s APIαʔόʹͰ͖Δ͚ͩෛՙΛ͔ ͚ͨ͘ͳ͍

Slide 70

Slide 70 text

ݕ౼ɾ४උͨ͠ࣄʢϞχλϦϯάʣ • Reverse ProxyΞϓϦΛ։ൃͯ͠ɺk8sΫϥελʹσϓϩΠ • PodͷϝτϦΫεऩू͸Reverse ProxyΞϓϦܦ༝Ͱߦ͏

Slide 71

Slide 71 text

·ͱΊ • LINE ShopαʔϏεΛ͝঺հ͠·ͨ͠ • LINE ShopαʔϏεΞʔΩςΫνϟΛ͝঺հ͠·ͨ͠ • ArmeriaͱCentral Dogma • ϚΠΫϩαʔϏεʹ͓͍ͯݕ౼͕ඞཁͳࣄ • Distributed Tracing • Cascading FailureΛ๷͙ͨΊͷCircuit Breaker • Graceful DegradationΛߟྀͨ͠αʔϏε෼ׂ • Service Discovery

Slide 72

Slide 72 text

·ͱΊ • LINE ShopνʔϜͰͷSREͷऔΓ૊ΈΛ͝঺հ͠·ͨ͠ • αʔϏεͷ৴པੑͷ֊૚ • ϞχλϦϯά • ΠϯγσϯτରԠ • ϙετϞʔςϜ • ΩϟύγςΟϓϥϯχϯά

Slide 73

Slide 73 text

LINE DEVELOPER DAY 2020ͷ͝Ҋ಺ • ެࣜαΠτ: https://linedevday.linecorp.com/2020/ja • ೔ఔ: 2020/11/25~27ʢΦϯϥΠϯ։࠵ʣ • ߹ܭ150Ҏ্ͷηογϣϯ • શηογϣϯ೔ӳ௨༁ରԠ

Slide 74

Slide 74 text

We are hiring • LINE Fukuokaגࣜձࣾ • αʔόʔαΠυΤϯδχΞ https://linefukuoka.co.jp/ja/career/list/engineer/ development_engineer_server-side • LINEגࣜձࣾ • γχΞαʔόʔαΠυΤϯδχΞ/ίϯςϯπൢചϓϥοτϑΥʔϜ https://linecorp.com/ja/career/position/665 • Site Reliability Engineer/ίϯςϯπൢചϓϥοτϑΥʔϜ https://linecorp.com/ja/career/position/1535

Slide 75

Slide 75 text

͝ਗ਼ௌ͋Γ͕ͱ͏͍͟͝·ͨ͠ʂ