Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE at Cookpad
Search
rrreeeyyy
November 29, 2017
Technology
1
1.3k
SRE at Cookpad
#hbstudy の 79 回目で SRE の話をしました。
rrreeeyyy
November 29, 2017
Tweet
Share
More Decks by rrreeeyyy
See All by rrreeeyyy
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
200
An Efficient Incident Response Training with AI / SRE NEXT 2024 Sponsor Session
rrreeeyyy
1
4.2k
カンファレンスから見る SRE トレンド 2024 / SRE Trends from Conferences in 2024 #SRE_Findy
rrreeeyyy
4
2.3k
信頼性の育て方 / mackerel-meetup-15
rrreeeyyy
10
2.5k
SRE の歩き方・進め方 / sre-walk-through-procedure
rrreeeyyy
0
8.6k
「信頼性」を保ちつつ大規模サービスをリニューアルする / cookpad-tech-kitchen-service-embedded-sres
rrreeeyyy
11
12k
Cookpad and Prometheus
rrreeeyyy
6
20k
SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview
rrreeeyyy
5
5.4k
A survey of anomaly detection methodologies for web system
rrreeeyyy
5
1.3k
Other Decks in Technology
See All in Technology
1行のコードから社会課題の解決へ: EMの探究、事業・技術・組織を紡ぐ実践知 / EM Conf 2025
9ma3r
6
2.2k
Autonomous Database Serverless 技術詳細 / adb-s_technical_detail_jp
oracle4engineer
PRO
17
45k
OpenID Connect for Identity Assurance の概要と翻訳版のご紹介 / 20250219-BizDay17-OIDC4IDA-Intro
oidfj
0
470
実は強い 非ViTな画像認識モデル
tattaka
1
1.1k
利用終了したドメイン名の最強終活〜観測環境を育てて、分析・供養している件〜 / The Ultimate End-of-Life Preparation for Discontinued Domain Names
nttcom
2
360
人はなぜISUCONに夢中になるのか
kakehashi
PRO
6
1.8k
AWSを活用したIoTにおけるセキュリティ対策のご紹介
kwskyk
0
290
ウォンテッドリーのデータパイプラインを支える ETL のための analytics, rds-exporter / analytics, rds-exporter for ETL to support Wantedly's data pipeline
unblee
0
110
Cracking the Coding Interview 6th Edition
gdplabs
14
28k
2025-02-21 ゆるSRE勉強会 Enhancing SRE Using AI
yoshiiryo1
1
460
次世代KYC活動報告 / 20250219-BizDay17-KYC-nextgen
oidfj
0
460
Active Directory攻防
cryptopeg
PRO
8
5.1k
Featured
See All Featured
Fashionably flexible responsive web design (full day workshop)
malarkey
406
66k
A designer walks into a library…
pauljervisheath
205
24k
Code Reviewing Like a Champion
maltzj
521
39k
Making the Leap to Tech Lead
cromwellryan
133
9.1k
Agile that works and the tools we love
rasmusluckow
328
21k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.7k
Become a Pro
speakerdeck
PRO
26
5.2k
Practical Orchestrator
shlominoach
186
10k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
59k
Producing Creativity
orderedlist
PRO
344
40k
Building Flexible Design Systems
yeseniaperezcruz
328
38k
Statistics for Hackers
jakevdp
797
220k
Transcript
SRE at Cookpad ΫοΫύουגࣜձࣾ ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ ٢ ཽଠ (
@rrreeeyyy ) hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 1
Agenda • SRE ʹ͍ͭͯ(վΊͯ) • ΫοΫύουʹ͓͚ΔΠϯϑϥߏ • EC2 with spot
instances • hako + ECS ڥ • ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ • ࣄͷྲྀΕɾऔΓΈʹ͍ͭͯɹ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 2
!SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB3ZPUB Me • Yoshikawa Ryota ( @rrreeeyyy
[reɪ] ) • גࣜձࣾϋʔτϏʔπ (2010/11 ʙ 2016/12) • ΫοΫύουגࣜձࣾ (2017/01 ʙ) • ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ • ڵຯྖҬ • ϞχλϦϯάɾ࣌ܥྻσʔλϕʔε • ࢄγεςϜɾϩʔυόϥϯα • झຯ • League of Legends, ΑΑΫϩχΫϧ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 3
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 4
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 5
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 6
(վΊͯ) Site Reliability Engineering • αΠτͷ৴པੑͷશͯʹൣғΛ࣋ͭ • ಛʹɺιϑτΣΞɾγεςϜΤϯδχΞϦϯάΛ༻͍ͯ՝Λղܾ͢Δ • ྫ͑ɺύϑΥʔϚϯε
• ύϑΥʔϚϯεʹ͕͋Δίʔυ SRE ͕ࣗίʔυΛॻ͍ͯमਖ਼͢Δ • ྫ͑ɺεέʔϥϏϦςΟɾΞϕΠϥϏϦςΟ • ਓؒʹΑΔखಈͷ࡞ۀΛίʔυʹΑΓࣗಈԽ͢Δ • ଐਓੑΛഉআͯ҆͠ఆͤ͞ΔɾਓؒΛ૿͞ͳͯ͘αʔϏεΛεέʔϧͤ͞Δ • ن͕૿େͯ҆͠ఆͯ͠αʔϏεΛఏڙͰ͖Δঢ়ଶʹ͢Δ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 7
SRE ͷ׆ಈ SRE ͷ׆ಈେ͖͚ͯ࣍͘ͷ 4 ͭʹྨͰ͖Δͱ͞Ε͍ͯΔ • So#ware Engineering •
Systems Engineering • Toil • Overhead hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 8
SRE ͷ׆ಈ • So#ware Engineering • ίʔυͷ࡞मਖ਼, ͦΕʹؔ࿈͢ΔઃܭυΩϡϝϯςʔγϣϯͷ࡞ۀͳͲ • ࣗಈԽεΫϦϓτͷ࡞,
πʔϧϑϨʔϜϫʔΫͷ࡞ ... • αʔϏεͷ৴པੑΛߴΊΔػೳͷՃ ... • Systems Engineering • 1 ճͷ࡞ۀͰӬଓతͳվળΛੜΉ࣍ͷΑ͏ͳ࡞ۀ • ຊ൪γεςϜͷઃఆมߋ ... • ։ൃνʔϜͷαʔϏεΞʔΩςΫνϟઃܭ ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 9
SRE ͷ׆ಈ • Toil • αʔϏεΛՔಇͤ͞ΔͨΊ͚ͩʹ܁Γฦ͠ߦΘΕΔखಈͰͷ࡞ۀ • ظతͳՁΛ࣋ͨͳ͍, αʔϏεͷʹରͯ͠ O(n)
Ͱ͋Δ ... • ͭ·Βͳ͍ࣄͰظతͳՁ͕͋Ε Toil Ͱͳ͍ • Overhead • αʔϏεΛՔಇͤ͞Δ͜ͱʹ݁͠ͳ͍ཧతͳ࡞ۀ • ࠾༻, ϛʔςΟϯά, ධՁ, τϨʔχϯά ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 10
ΫοΫύουʹ͓͚Δ SRE hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy )
11
ΫοΫύουʹ͓͚Δ SRE • ΠϯϑϥετϥΫνϟʔ෦ • ͷதʹ SRE άϧʔϓ͕͋Δ • DWH,
Security, ࣾIT ͕ಉ͡෦ॺʹ͋Δ • ϝϯόʔ SRE ͚ͩͰ 10 ਓఔ + ւ֎ʹ 2 ໊ఔ • ࠃ 7 ׂɾάϩʔόϧ 1 3 ׂ͙Β͍ͷൺ 1 h$ps:/ /speakerdeck.com/sorah/building-infrastructure-for-our-global-service hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 12
ΫοΫύουʹ͓͚ΔΠϯϑϥߏ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 13
ΫοΫύουʹ͓͚ΔΠϯϑϥߏ • ༷ʑ͋Δ͕ϝΠϯͰಈ͍͍ͯΔͷେ͖͚ͯ͘ 2 ͭ • EC2 + spotscaler ڥ
• EC2 ʹΞϓϦέʔγϣϯΛσϓϩΠ͍ͯ͠Δߏ • EC2 Πϯελϯεεϙοτʹͳ͍ͬͯΔ • ECS + hako ڥ • ECS Ϋϥελʹରͯ͠ίϯςφΛσϓϩΠ͍ͯ͠Δߏ • σϓϩΠϝϯτʹ eagletmt/hako Λར༻͍ͯ͠Δ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 14
EC2 + spotscaler ڥ • ΫοΫύουͷ͍ΘΏΔը໘ͷඳը෦ͳͲ͕͜Εʹ͋ͨΔ • PC ൛αΠτ, εϚϗ༻ͷ
API ... ͳͲ • Route53 + ALB + EC2 ͷ AWS ͷҰൠతͳߏ • EC2 spot instance Λ spotscaler 2 Λͬͯཧ͍ͯ͠Δ • deployment serf + sorah/mamiya 3 Λͬͯߦ͍ͬͯΔ 3 h$ps:/ /github.com/eagletmt/hako 2 h$ps:/ /speakerdeck.com/ryotarai/spot-instances-in-cookpad-number-cookpadtechconf-2017 hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 15
ECS + hako ڥ • ΫοΫύουͷ༷ʑͳϚΠΫϩαʔϏε͕͜Εʹ͋ͨΔ • λΠϜϥΠϯ, ࠂ, ೝূ...
ͳͲ • ػցֶशͷج൫͜͜ʹ͍ͬͯΔʢg2 Πϯελϯεͷ ECS Ϋϥελʣ • ECS ͷΫϥελʹରͯ͠ hako 3 Λར༻ͯ͠σϓϩΠ͍ͯ͠Δ • GHE ্ͷϦϙδτϦʹΞϓϦέʔγϣϯఆٛͷ PR Λग़ͯ͠Β͏ • σϓϩΠ Slack ্͔Β։ൃऀ͕جຊతʹࣗ༝ʹߦ͑Δ • PR ຖͷεςʔδϯάɾ౷߹ίϯιʔϧɾτϨʔγϯάͳͲ͕͋Δ 4 4 h$p:/ /techlife.cookpad.com/entry/2016/03/16/100043 3 h$ps:/ /github.com/eagletmt/hako hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 16
ECS + hako ڥ ʹ͓͚Δ YAML ઃఆ scheduler: <<: !include
schedulers/default.yml elb_v2: vpc_id: vpc-xxxxxx health_check_path: /healthcheck listeners: - port: 80 protocol: HTTP # : autoscaling: min_capacity: 2 max_capacity: 24 policies: # : policies for autoscaling hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 17
ECS + hako ڥ ʹ͓͚Δ YAML ઃఆ app: image: xxxxxxxxxx/service
# ECR Λࢦఆ͢Δ͜ͱ͕ଟ͍ cpu: 128 memory: 512 links: - fluentd env: $providers: - <<: !include env_providers/vault.yml directory: hako/group/service # vault hako/group/service ͷ - !include env_providers/vault_shared.yml # ֘ͷάϧʔϓʹॴଐ͢Δਓ͕ࣗ༝ʹઃఆͰ͖Δ RAILS_ENV: production WORKER_NUM: '2' additional_containers: front: !include containers/front.yml fluentd: !include containers/fluentd.yml volumes: td-agent: source_path: /var/lib/hako/td-agent/service hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 18
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 19
ΫοΫύουʹ͓͚ΔϚΠΫϩαʔϏεͷ࣮ફ • ৫ਓɾνʔϜɾαʔϏεͷʹͭΕͯݖݶͷҕৡ͕ඞཁʹͳͬͨ • SRE ج൫νʔϜ͕શͯΛཧ͢ΔͷෆՄೳͩ͠ෆ݈શ • ΞϓϦέʔγϣϯࣗମ͕େ͖͘ͳΓ༷͗ͯ͢ʑͳ͕ग़ΔΑ͏ʹͳͬͨ • ίʔυͷݟ௨͕͠ѱ͘ͳΔɾςετʹֻ͕͔࣌ؒΓ͗͢Δ
... • લ͔ΒঃʑʹඋΛਐΊ͍ͯΔ 4 4 h$p:/ /techlife.cookpad.com/entry/2016/03/16/100043 hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 20
ΫοΫύουʹ͓͚ΔϚΠΫϩαʔϏεͷ࣮ફ • جຊతʹϝΠϯͷΞϓϦέʔγϣϯΛ hako ڥʹΓग़͢ • ৽نʹ࡞͞ΕΔΞϓϦέʔγϣϯ hako ڥΛඪ४ʹ͢Δ •
ಉ࣌ʹɺmicroservice ԽʹΑΔ՝ʹ͍͔ͭ͘औΓΜͰ͍Δ • αʔϏεؒͷ Tracing ʹ AWS X-Ray Λར༻͍ͯ͠Δ • αʔϏεϝογϡͷ࣮ݱʹ envoyproxy/envoy Λར༻͍ͯ͠Δ • ͍͔ͭ͘՝͕͋Γ࣮ͭͭࡍʹಈ͔ͯ͠ΛਐΊ͍ͯΔ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 21
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 22
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 23
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy
) 24
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ • ΫοΫύουͷ SRE ͷ׆ಈΛ͍͔ͭ͘հ • αʔόߏͷཧɾมߋ •
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • ϞχλϦϯάͷվળ • ϩʔυόϥϯγϯάͷվળ • ৽ଔݚम • ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 25
αʔόߏͷมߋɾཧ • αʔόߏجຊతʹશͯίʔυԽ͞Ε͍ͯΔ • Codenize.tools 5 Itamae 6, Terraform
ͳͲ • IAM User Route53 GHE ্Ͱී௨ʹཧ͞Ε͍ͯΔ • ։ൃऀ͕ࣗ༝ʹ PR Λग़͢͜ͱ͕ग़དྷΔ • αʔόߏͷมߋίʔυͷ Pull request ͱͯ͠ग़͢ • ϨϏϡʔΛड͚ɺϚʔδ͔ͯ͠Βө • Ұ෦Ϛʔδͯࣗ͠ಈͰө͞ΕΔ 6 h$ps:/ /github.com/itamae-kitchen/itamae 5 h$ps:/ /codenize.tools/ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 26
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 27
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy
) 28
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 29
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • େ͖ΊͳػೳαʔϏε୯ҐͰׂͯ͠։ൃ͢Δ • ߹ʹΑΔ͕ SRE ͕ 1
ਓΞαΠϯ͞ΕΔ • ֘ͷαʔϏεͷ৴པੑʹؔ͢Δ͜ͱجຊతʹશ෦Δ • ઃܭͷ૬ஊɾϨϏϡʔ • ϦϦʔεલͷෛՙࢼݧɾΩϟύγςΟϓϥϯχϯά • ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν • ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 30
ઃܭͷ૬ஊɾϨϏϡʔ • ։ൃͷલʹҎԼͷΑ͏ͳ͜ͱΛڞ༗ͯ͠Β͍ٞ͢Δ • ͲͷΑ͏ͳΞϓϦέʔγϣϯΛ࡞Δ͔ • ͲͷΑ͏ͳίϯϙʔωϯτ͕ඞཁ͔ • ͲͷΑ͏ͳߏɾεέʔϧઓུʹ͢Δ͔ •
Design Document Λॻ͍ͯΒ͍ͦΕΛϕʔεʹΛͨ͠Γ͢Δ • େ͖ΊͷػೳͳͲΛ։ൃ͢Δ࣌ Design Document Λॻ͘͜ͱ͕ଟ͍ (SRE ) hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 31
ෛՙࢼݧ • ͋Δఔग़དྷ͖ͯͨลΓͰෛՙࢼݧͷΛ͢Δ • Ͳͷ͘Β͍ͷϦΫΤετ͕ఆ͞ΕΔ͔ • ͲͷΑ͏ͳϢʔβϦΫΤετ͕ఆ͞ΕΔ͔ • SRE ଆͰෛՙࢼݧͷγφϦΦڥΛ࡞Δ
• ෛՙࢼݧΛߦ͍ͳ͕Βຊ൪ڥͰͷΩϟύγςΟΛݟੵΔ • ΞϓϦέʔγϣϯ্ͷϘτϧωοΫ͕Ͳ͜ʹ͋Δ͔ΛݟΔ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 32
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 33
ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν • ৴པੑʹؔ͢Δ͕͋Δ߹ࣗͰίʔυΛॻ͘ • ྫ͑ύϑΥʔϚϯε্ͷͳͲ • ΞϓϦέʔγϣϯͷ༷Λཧղͯ͠దͳΞϓϩʔνΛऔΔ • ྫ͑Ͳ͏͍͏ը໘ͷͲ͏͍͏ΫΤϦͳͷ͔ •
Ͳ͏͍͏ใ͕ཉͯ͘͠Ͳ͏͍͏੍͕͋Δͷ͔ • ͍ͬͯΔϛυϧΣΞͷ༷ཧղͯ͠దͳΞϓϩʔνΛऔΔ • ྫ͑ SQL ͷվળͳΒ MySQL ͷΠϯσοΫεͷߏ෦ॲཧͷ͕ࣝ͋Δͱྑ͍ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 34
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 35
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • ͜ͷลΓͨΓલͷࣄΛී௨ʹ͍ͬͯΔ͚ͩ • ͬͱվળग़དྷΔՕॴ͕ͨ͘͞Μ͋Δͱߟ͍͑ͯΔ • ྫ͑ෛՙࢼݧΛࣗಈͰߦ͏Α͏ʹ͢ΔɺͳͲ •
ΞϓϦέʔγϣϯΛཧղ͠ɺඞཁͳΒίʔυΛॻ͘ • SRE αΠτͷ৴པੑʹؔΘΔશͯʹΛෛ͏ • ΞϓϦέʔγϣϯͷಛੑઃܭΛཧղ͠ϛυϧΣΞΛΈ߹ΘͤΔ Systems Engineering • ༷ϛυϧΣΞΛཧղ࣮͠ࡍʹΞϓϦέʔγϣϯͷϩδοΫΛॻ͘ So.ware Engineering hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 36
ϞχλϦϯάͷվળ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 37
ϞχλϦϯάͷվળ • ݱࡏϝτϦΫεऔಘʹ Zabbix Λ༻͍ͯ͠Δ • ΑΓεέʔϧ͠ɺߴڃݴޠͰϝτϦΫεΞϥʔτΛѻ͑ΔΑ͏ʹ͍ͨ͠ • SRE ຊ
ͷ 10 ষʹॻ͔Ε͍ͯΔΑ͏ͳঢ়ଶʹͳΓ͍ͨ • Prometheus ͷಋೖΛঃʑʹ࣮ࢪ͍ͯ͠Δ • ϝτϦΫεͷղ૾ 15 ඵ୯ҐͳͲͰऔಘ͢ΔΑ͏ʹ͍ͯ͠Δ • ୯७ͳ HTTP ϨεϙϯεͰϝτϦΫεΛऩूग़དྷΔ • ΞϓϦέʔγϣϯࣗମͷϝτϦΫεͳͲ͕ൺֱత؆୯ʹऩूग़དྷΔ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 38
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 39
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 40
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 41
ϞχλϦϯάͷվળ • Ͳ͏͍ͬͨϛυϧΣΞΛબఆ͢Δ͔ • ϞχλϦϯάͰ͋Ε࣌ܥྻσʔλϕʔεͳͲʹؔ͢Δਂ͍ཧղΛ͔ͯ͠ΒબͿ • Ͳ͏͍ͬͨಛੑ͕͋Δ͔ʁࣗͨͪͷنʹ߹͍ͬͯΔ͔ʁεέʔϧ͢Δ͔ʁ • Γͳ͍ίϯϙʔωϯτͳ͍͔ʁແ͍ͳΒࣗ࡞Ͱ͖ͦ͏ͳن͔ʁ •
Prometheus ͳΒྫ͑ Long-Term Storage पΓͷίϯϙʔωϯτͳͲ • ͔ͬ͠Γͱͨ͠ Systems Engineering ͷݟΛར༻ͯ͠બఆ͢Δ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 42
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 43
ϩʔυόϥϯγϯάͷվળ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 44
ϩʔυόϥϯγϯάͷվળ • ैདྷΞϓϦέʔγϣϯαʔόͷϩʔυόϥϯγϯάʹ haproxy Λར༻͍ͯͨ͠ • Weighted Round-robin (wrr) Λར༻ͯ͠όϥϯγϯάΛߦ͍ͬͯͨ
• ෛՙͷ͍ΤϯυϙΠϯτͱߴ͍ΤϯυϙΠϯτ͕͋Δ߹ෛՙ͕Βͭ͘ • wrr ͩͱಛఆͷαʔόʹෛՙͷߴ͍ΤϯυϙΠϯτ͕ूத͢ΔՄೳੑ͕͋Δ • leastconn Λ͑ྑͦ͞͏͕ͩ HTTP keepalive ͍ͯ͠Δ߹ʹ͕͋Δ • ίωΫγϣϯ͕ΔͷͰ࣮ࡍʹॲཧ͍ͯ͠ΔϦΫΤετͱണ͢Δɹ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 45
ryotarai/simproxy • h#ps:/ /github.com/ryotarai/simproxy • Go Ͱॻ͔ΕͨγϯϓϧͳϦόʔεϓϩΩγ • ϔϧενΣοΫͳͲͷجຊతͳػೳ͋Δ •
Balancing Method ʹ leastreq Λ࣮͍ͯ͠Δ • ࢦඪͱͯ͠୯७ͳίωΫγϣϯͰͳ࣍͘ͷΑ͏ͳΛར༻͢Δ • όοΫΤϯυ͕ड͚औ͕ͬͨԠ͕ฦ͖͍ͬͯͯͳ͍ϦΫΤετͷ • ͜ΕʹΑΓ HTTP keepalive ͕͋ͬͯ leastconn ͷΑ͏ͳόϥϯγϯά͕Մೳ • ϩʔυόϥϯγϯάͷΞϧΰϦζϜʹؔ͢Δ Systems Engineering ͱ࣮͢Δ So.ware Engineering hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 46
before simproxy hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy )
47
a"er simproxy hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy )
48
৽ଔݚम • ৽ଔʹରͯ͠ΠϯϑϥͷݚमΛ 3 ؒߦ͍ͬͯΔ • MySQL ͷΠϯσοΫεͷߏΛཧղ͠ΫΤϦνϡʔχϯάΛߦ͏ ... •
AWS ͷ ALB RDS ͳͲΛ͍ࣗͷΞϓϦέʔγϣϯΛεέʔϧͤ͞Δ ... • ࣮ࡍͷΞϓϦέʔγϣϯΛॻ͘ϝϯόʔ͕Πϯϑϥͷཧղ͕͋Δํ͕ྑ͍ • શͯͷΞϓϦέʔγϣϯίʔυΛ SRE ͕νΣοΫ͍ͯ͠Δͱεέʔϧ͠ͳ͍ • ྨ্ Overhead ͔͠Εͳ͍͕αʔϏεΛεέʔϧͤ͞ΔͨΊʹඞཁ • ࣮ࡍʹ Systems Engineering ͷਂ͍ݟ͕ແ͍ͱڭ͑Δ͜ͱ͕ग़དྷͳ͍ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 49
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 50
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 51
PR hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 52
We are hiring! • ΫοΫύουͰຊհͨ͠Α͏ͳऔΓΈΛߦ͍ͬͯ·͢ • ·ͩ·ͩղܾ͍ͨ͠ɾऔΓΈ͍ͨ՝͕ͨ͘͞Μ͋Δ • ָ͠Έͳ͕ΒҰॹʹΕΔϝϯόʔΛืू͍ͯ͠·͢ →
h$ps:/ /cookpad.jobs hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 53