Upgrade to PRO for Only $50/Year—Limited-Time Offer! 🔥
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE at Cookpad
Search
rrreeeyyy
November 29, 2017
Technology
1
1.4k
SRE at Cookpad
#hbstudy の 79 回目で SRE の話をしました。
rrreeeyyy
November 29, 2017
Tweet
Share
More Decks by rrreeeyyy
See All by rrreeeyyy
Rethinking Incident Response: Context-Aware AI in Practice - Incident Buddy Edition -
rrreeeyyy
0
180
Rethinking Incident Response: Context-Aware AI in Practice
rrreeeyyy
3
2.2k
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
260
An Efficient Incident Response Training with AI / SRE NEXT 2024 Sponsor Session
rrreeeyyy
1
5.7k
カンファレンスから見る SRE トレンド 2024 / SRE Trends from Conferences in 2024 #SRE_Findy
rrreeeyyy
4
2.5k
信頼性の育て方 / mackerel-meetup-15
rrreeeyyy
10
2.8k
SRE の歩き方・進め方 / sre-walk-through-procedure
rrreeeyyy
0
8.9k
「信頼性」を保ちつつ大規模サービスをリニューアルする / cookpad-tech-kitchen-service-embedded-sres
rrreeeyyy
11
13k
Cookpad and Prometheus
rrreeeyyy
6
21k
Other Decks in Technology
See All in Technology
AWSセキュリティアップデートとAWSを育てる話
cmusudakeisuke
0
280
re:Invent 2025 ~何をする者であり、どこへいくのか~
tetutetu214
0
210
エンジニアとPMのドメイン知識の溝をなくす、 AIネイティブな開発プロセス
applism118
4
1.3k
Oracle Cloud Infrastructure IaaS 新機能アップデート 2025/09 - 2025/11
oracle4engineer
PRO
0
120
技術以外の世界に『越境』しエンジニアとして進化を遂げる 〜Kotlinへの愛とDevHRとしての挑戦を添えて〜
subroh0508
1
450
Snowflakeでデータ基盤を もう一度作り直すなら / rebuilding-data-platform-with-snowflake
pei0804
5
1.5k
AWSを使う上で最低限知っておきたいセキュリティ研修を社内で実施した話 ~みんなでやるセキュリティ~
maimyyym
2
1k
mairuでつくるクレデンシャルレス開発環境 / Credential-less development environment using Mailru
mirakui
5
480
SSO方式とJumpアカウント方式の比較と設計方針
yuobayashi
7
670
AI駆動開発における設計思想 認知負荷を下げるフロントエンドアーキテクチャ/ 20251211 Teppei Hanai
shift_evolve
PRO
2
380
文字列の並び順 / Unicode Collation
tmtms
3
580
Databricks向けJupyter Kernelでデータサイエンティストの開発環境をAI-Readyにする / Data+AI World Tour Tokyo After Party
genda
1
120
Featured
See All Featured
Unsuck your backbone
ammeep
671
58k
Why You Should Never Use an ORM
jnunemaker
PRO
61
9.6k
Code Review Best Practice
trishagee
74
19k
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
359
30k
Documentation Writing (for coders)
carmenintech
76
5.2k
How GitHub (no longer) Works
holman
316
140k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
32
1.8k
Site-Speed That Sticks
csswizardry
13
1k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
254
22k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
12
1.3k
Practical Orchestrator
shlominoach
190
11k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Transcript
SRE at Cookpad ΫοΫύουגࣜձࣾ ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ ٢ ཽଠ (
@rrreeeyyy ) hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 1
Agenda • SRE ʹ͍ͭͯ(վΊͯ) • ΫοΫύουʹ͓͚ΔΠϯϑϥߏ • EC2 with spot
instances • hako + ECS ڥ • ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ • ࣄͷྲྀΕɾऔΓΈʹ͍ͭͯɹ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 2
!SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB3ZPUB Me • Yoshikawa Ryota ( @rrreeeyyy
[reɪ] ) • גࣜձࣾϋʔτϏʔπ (2010/11 ʙ 2016/12) • ΫοΫύουגࣜձࣾ (2017/01 ʙ) • ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ • ڵຯྖҬ • ϞχλϦϯάɾ࣌ܥྻσʔλϕʔε • ࢄγεςϜɾϩʔυόϥϯα • झຯ • League of Legends, ΑΑΫϩχΫϧ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 3
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 4
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 5
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 6
(վΊͯ) Site Reliability Engineering • αΠτͷ৴པੑͷશͯʹൣғΛ࣋ͭ • ಛʹɺιϑτΣΞɾγεςϜΤϯδχΞϦϯάΛ༻͍ͯ՝Λղܾ͢Δ • ྫ͑ɺύϑΥʔϚϯε
• ύϑΥʔϚϯεʹ͕͋Δίʔυ SRE ͕ࣗίʔυΛॻ͍ͯमਖ਼͢Δ • ྫ͑ɺεέʔϥϏϦςΟɾΞϕΠϥϏϦςΟ • ਓؒʹΑΔखಈͷ࡞ۀΛίʔυʹΑΓࣗಈԽ͢Δ • ଐਓੑΛഉআͯ҆͠ఆͤ͞ΔɾਓؒΛ૿͞ͳͯ͘αʔϏεΛεέʔϧͤ͞Δ • ن͕૿େͯ҆͠ఆͯ͠αʔϏεΛఏڙͰ͖Δঢ়ଶʹ͢Δ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 7
SRE ͷ׆ಈ SRE ͷ׆ಈେ͖͚ͯ࣍͘ͷ 4 ͭʹྨͰ͖Δͱ͞Ε͍ͯΔ • So#ware Engineering •
Systems Engineering • Toil • Overhead hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 8
SRE ͷ׆ಈ • So#ware Engineering • ίʔυͷ࡞मਖ਼, ͦΕʹؔ࿈͢ΔઃܭυΩϡϝϯςʔγϣϯͷ࡞ۀͳͲ • ࣗಈԽεΫϦϓτͷ࡞,
πʔϧϑϨʔϜϫʔΫͷ࡞ ... • αʔϏεͷ৴པੑΛߴΊΔػೳͷՃ ... • Systems Engineering • 1 ճͷ࡞ۀͰӬଓతͳվળΛੜΉ࣍ͷΑ͏ͳ࡞ۀ • ຊ൪γεςϜͷઃఆมߋ ... • ։ൃνʔϜͷαʔϏεΞʔΩςΫνϟઃܭ ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 9
SRE ͷ׆ಈ • Toil • αʔϏεΛՔಇͤ͞ΔͨΊ͚ͩʹ܁Γฦ͠ߦΘΕΔखಈͰͷ࡞ۀ • ظతͳՁΛ࣋ͨͳ͍, αʔϏεͷʹରͯ͠ O(n)
Ͱ͋Δ ... • ͭ·Βͳ͍ࣄͰظతͳՁ͕͋Ε Toil Ͱͳ͍ • Overhead • αʔϏεΛՔಇͤ͞Δ͜ͱʹ݁͠ͳ͍ཧతͳ࡞ۀ • ࠾༻, ϛʔςΟϯά, ධՁ, τϨʔχϯά ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 10
ΫοΫύουʹ͓͚Δ SRE hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy )
11
ΫοΫύουʹ͓͚Δ SRE • ΠϯϑϥετϥΫνϟʔ෦ • ͷதʹ SRE άϧʔϓ͕͋Δ • DWH,
Security, ࣾIT ͕ಉ͡෦ॺʹ͋Δ • ϝϯόʔ SRE ͚ͩͰ 10 ਓఔ + ւ֎ʹ 2 ໊ఔ • ࠃ 7 ׂɾάϩʔόϧ 1 3 ׂ͙Β͍ͷൺ 1 h$ps:/ /speakerdeck.com/sorah/building-infrastructure-for-our-global-service hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 12
ΫοΫύουʹ͓͚ΔΠϯϑϥߏ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 13
ΫοΫύουʹ͓͚ΔΠϯϑϥߏ • ༷ʑ͋Δ͕ϝΠϯͰಈ͍͍ͯΔͷେ͖͚ͯ͘ 2 ͭ • EC2 + spotscaler ڥ
• EC2 ʹΞϓϦέʔγϣϯΛσϓϩΠ͍ͯ͠Δߏ • EC2 Πϯελϯεεϙοτʹͳ͍ͬͯΔ • ECS + hako ڥ • ECS Ϋϥελʹରͯ͠ίϯςφΛσϓϩΠ͍ͯ͠Δߏ • σϓϩΠϝϯτʹ eagletmt/hako Λར༻͍ͯ͠Δ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 14
EC2 + spotscaler ڥ • ΫοΫύουͷ͍ΘΏΔը໘ͷඳը෦ͳͲ͕͜Εʹ͋ͨΔ • PC ൛αΠτ, εϚϗ༻ͷ
API ... ͳͲ • Route53 + ALB + EC2 ͷ AWS ͷҰൠతͳߏ • EC2 spot instance Λ spotscaler 2 Λͬͯཧ͍ͯ͠Δ • deployment serf + sorah/mamiya 3 Λͬͯߦ͍ͬͯΔ 3 h$ps:/ /github.com/eagletmt/hako 2 h$ps:/ /speakerdeck.com/ryotarai/spot-instances-in-cookpad-number-cookpadtechconf-2017 hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 15
ECS + hako ڥ • ΫοΫύουͷ༷ʑͳϚΠΫϩαʔϏε͕͜Εʹ͋ͨΔ • λΠϜϥΠϯ, ࠂ, ೝূ...
ͳͲ • ػցֶशͷج൫͜͜ʹ͍ͬͯΔʢg2 Πϯελϯεͷ ECS Ϋϥελʣ • ECS ͷΫϥελʹରͯ͠ hako 3 Λར༻ͯ͠σϓϩΠ͍ͯ͠Δ • GHE ্ͷϦϙδτϦʹΞϓϦέʔγϣϯఆٛͷ PR Λग़ͯ͠Β͏ • σϓϩΠ Slack ্͔Β։ൃऀ͕جຊతʹࣗ༝ʹߦ͑Δ • PR ຖͷεςʔδϯάɾ౷߹ίϯιʔϧɾτϨʔγϯάͳͲ͕͋Δ 4 4 h$p:/ /techlife.cookpad.com/entry/2016/03/16/100043 3 h$ps:/ /github.com/eagletmt/hako hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 16
ECS + hako ڥ ʹ͓͚Δ YAML ઃఆ scheduler: <<: !include
schedulers/default.yml elb_v2: vpc_id: vpc-xxxxxx health_check_path: /healthcheck listeners: - port: 80 protocol: HTTP # : autoscaling: min_capacity: 2 max_capacity: 24 policies: # : policies for autoscaling hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 17
ECS + hako ڥ ʹ͓͚Δ YAML ઃఆ app: image: xxxxxxxxxx/service
# ECR Λࢦఆ͢Δ͜ͱ͕ଟ͍ cpu: 128 memory: 512 links: - fluentd env: $providers: - <<: !include env_providers/vault.yml directory: hako/group/service # vault hako/group/service ͷ - !include env_providers/vault_shared.yml # ֘ͷάϧʔϓʹॴଐ͢Δਓ͕ࣗ༝ʹઃఆͰ͖Δ RAILS_ENV: production WORKER_NUM: '2' additional_containers: front: !include containers/front.yml fluentd: !include containers/fluentd.yml volumes: td-agent: source_path: /var/lib/hako/td-agent/service hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 18
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 19
ΫοΫύουʹ͓͚ΔϚΠΫϩαʔϏεͷ࣮ફ • ৫ਓɾνʔϜɾαʔϏεͷʹͭΕͯݖݶͷҕৡ͕ඞཁʹͳͬͨ • SRE ج൫νʔϜ͕શͯΛཧ͢ΔͷෆՄೳͩ͠ෆ݈શ • ΞϓϦέʔγϣϯࣗମ͕େ͖͘ͳΓ༷͗ͯ͢ʑͳ͕ग़ΔΑ͏ʹͳͬͨ • ίʔυͷݟ௨͕͠ѱ͘ͳΔɾςετʹֻ͕͔࣌ؒΓ͗͢Δ
... • લ͔ΒঃʑʹඋΛਐΊ͍ͯΔ 4 4 h$p:/ /techlife.cookpad.com/entry/2016/03/16/100043 hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 20
ΫοΫύουʹ͓͚ΔϚΠΫϩαʔϏεͷ࣮ફ • جຊతʹϝΠϯͷΞϓϦέʔγϣϯΛ hako ڥʹΓग़͢ • ৽نʹ࡞͞ΕΔΞϓϦέʔγϣϯ hako ڥΛඪ४ʹ͢Δ •
ಉ࣌ʹɺmicroservice ԽʹΑΔ՝ʹ͍͔ͭ͘औΓΜͰ͍Δ • αʔϏεؒͷ Tracing ʹ AWS X-Ray Λར༻͍ͯ͠Δ • αʔϏεϝογϡͷ࣮ݱʹ envoyproxy/envoy Λར༻͍ͯ͠Δ • ͍͔ͭ͘՝͕͋Γ࣮ͭͭࡍʹಈ͔ͯ͠ΛਐΊ͍ͯΔ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 21
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 22
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 23
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy
) 24
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ • ΫοΫύουͷ SRE ͷ׆ಈΛ͍͔ͭ͘հ • αʔόߏͷཧɾมߋ •
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • ϞχλϦϯάͷվળ • ϩʔυόϥϯγϯάͷվળ • ৽ଔݚम • ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 25
αʔόߏͷมߋɾཧ • αʔόߏجຊతʹશͯίʔυԽ͞Ε͍ͯΔ • Codenize.tools 5 Itamae 6, Terraform
ͳͲ • IAM User Route53 GHE ্Ͱී௨ʹཧ͞Ε͍ͯΔ • ։ൃऀ͕ࣗ༝ʹ PR Λग़͢͜ͱ͕ग़དྷΔ • αʔόߏͷมߋίʔυͷ Pull request ͱͯ͠ग़͢ • ϨϏϡʔΛड͚ɺϚʔδ͔ͯ͠Βө • Ұ෦Ϛʔδͯࣗ͠ಈͰө͞ΕΔ 6 h$ps:/ /github.com/itamae-kitchen/itamae 5 h$ps:/ /codenize.tools/ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 26
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 27
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy
) 28
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 29
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • େ͖ΊͳػೳαʔϏε୯ҐͰׂͯ͠։ൃ͢Δ • ߹ʹΑΔ͕ SRE ͕ 1
ਓΞαΠϯ͞ΕΔ • ֘ͷαʔϏεͷ৴པੑʹؔ͢Δ͜ͱجຊతʹશ෦Δ • ઃܭͷ૬ஊɾϨϏϡʔ • ϦϦʔεલͷෛՙࢼݧɾΩϟύγςΟϓϥϯχϯά • ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν • ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 30
ઃܭͷ૬ஊɾϨϏϡʔ • ։ൃͷલʹҎԼͷΑ͏ͳ͜ͱΛڞ༗ͯ͠Β͍ٞ͢Δ • ͲͷΑ͏ͳΞϓϦέʔγϣϯΛ࡞Δ͔ • ͲͷΑ͏ͳίϯϙʔωϯτ͕ඞཁ͔ • ͲͷΑ͏ͳߏɾεέʔϧઓུʹ͢Δ͔ •
Design Document Λॻ͍ͯΒ͍ͦΕΛϕʔεʹΛͨ͠Γ͢Δ • େ͖ΊͷػೳͳͲΛ։ൃ͢Δ࣌ Design Document Λॻ͘͜ͱ͕ଟ͍ (SRE ) hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 31
ෛՙࢼݧ • ͋Δఔग़དྷ͖ͯͨลΓͰෛՙࢼݧͷΛ͢Δ • Ͳͷ͘Β͍ͷϦΫΤετ͕ఆ͞ΕΔ͔ • ͲͷΑ͏ͳϢʔβϦΫΤετ͕ఆ͞ΕΔ͔ • SRE ଆͰෛՙࢼݧͷγφϦΦڥΛ࡞Δ
• ෛՙࢼݧΛߦ͍ͳ͕Βຊ൪ڥͰͷΩϟύγςΟΛݟੵΔ • ΞϓϦέʔγϣϯ্ͷϘτϧωοΫ͕Ͳ͜ʹ͋Δ͔ΛݟΔ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 32
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 33
ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν • ৴པੑʹؔ͢Δ͕͋Δ߹ࣗͰίʔυΛॻ͘ • ྫ͑ύϑΥʔϚϯε্ͷͳͲ • ΞϓϦέʔγϣϯͷ༷Λཧղͯ͠దͳΞϓϩʔνΛऔΔ • ྫ͑Ͳ͏͍͏ը໘ͷͲ͏͍͏ΫΤϦͳͷ͔ •
Ͳ͏͍͏ใ͕ཉͯ͘͠Ͳ͏͍͏੍͕͋Δͷ͔ • ͍ͬͯΔϛυϧΣΞͷ༷ཧղͯ͠దͳΞϓϩʔνΛऔΔ • ྫ͑ SQL ͷվળͳΒ MySQL ͷΠϯσοΫεͷߏ෦ॲཧͷ͕ࣝ͋Δͱྑ͍ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 34
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 35
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • ͜ͷลΓͨΓલͷࣄΛී௨ʹ͍ͬͯΔ͚ͩ • ͬͱվળग़དྷΔՕॴ͕ͨ͘͞Μ͋Δͱߟ͍͑ͯΔ • ྫ͑ෛՙࢼݧΛࣗಈͰߦ͏Α͏ʹ͢ΔɺͳͲ •
ΞϓϦέʔγϣϯΛཧղ͠ɺඞཁͳΒίʔυΛॻ͘ • SRE αΠτͷ৴པੑʹؔΘΔશͯʹΛෛ͏ • ΞϓϦέʔγϣϯͷಛੑઃܭΛཧղ͠ϛυϧΣΞΛΈ߹ΘͤΔ Systems Engineering • ༷ϛυϧΣΞΛཧղ࣮͠ࡍʹΞϓϦέʔγϣϯͷϩδοΫΛॻ͘ So.ware Engineering hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 36
ϞχλϦϯάͷվળ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 37
ϞχλϦϯάͷվળ • ݱࡏϝτϦΫεऔಘʹ Zabbix Λ༻͍ͯ͠Δ • ΑΓεέʔϧ͠ɺߴڃݴޠͰϝτϦΫεΞϥʔτΛѻ͑ΔΑ͏ʹ͍ͨ͠ • SRE ຊ
ͷ 10 ষʹॻ͔Ε͍ͯΔΑ͏ͳঢ়ଶʹͳΓ͍ͨ • Prometheus ͷಋೖΛঃʑʹ࣮ࢪ͍ͯ͠Δ • ϝτϦΫεͷղ૾ 15 ඵ୯ҐͳͲͰऔಘ͢ΔΑ͏ʹ͍ͯ͠Δ • ୯७ͳ HTTP ϨεϙϯεͰϝτϦΫεΛऩूग़དྷΔ • ΞϓϦέʔγϣϯࣗମͷϝτϦΫεͳͲ͕ൺֱత؆୯ʹऩूग़དྷΔ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 38
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 39
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 40
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 41
ϞχλϦϯάͷվળ • Ͳ͏͍ͬͨϛυϧΣΞΛબఆ͢Δ͔ • ϞχλϦϯάͰ͋Ε࣌ܥྻσʔλϕʔεͳͲʹؔ͢Δਂ͍ཧղΛ͔ͯ͠ΒબͿ • Ͳ͏͍ͬͨಛੑ͕͋Δ͔ʁࣗͨͪͷنʹ߹͍ͬͯΔ͔ʁεέʔϧ͢Δ͔ʁ • Γͳ͍ίϯϙʔωϯτͳ͍͔ʁແ͍ͳΒࣗ࡞Ͱ͖ͦ͏ͳن͔ʁ •
Prometheus ͳΒྫ͑ Long-Term Storage पΓͷίϯϙʔωϯτͳͲ • ͔ͬ͠Γͱͨ͠ Systems Engineering ͷݟΛར༻ͯ͠બఆ͢Δ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 42
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 43
ϩʔυόϥϯγϯάͷվળ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 44
ϩʔυόϥϯγϯάͷվળ • ैདྷΞϓϦέʔγϣϯαʔόͷϩʔυόϥϯγϯάʹ haproxy Λར༻͍ͯͨ͠ • Weighted Round-robin (wrr) Λར༻ͯ͠όϥϯγϯάΛߦ͍ͬͯͨ
• ෛՙͷ͍ΤϯυϙΠϯτͱߴ͍ΤϯυϙΠϯτ͕͋Δ߹ෛՙ͕Βͭ͘ • wrr ͩͱಛఆͷαʔόʹෛՙͷߴ͍ΤϯυϙΠϯτ͕ूத͢ΔՄೳੑ͕͋Δ • leastconn Λ͑ྑͦ͞͏͕ͩ HTTP keepalive ͍ͯ͠Δ߹ʹ͕͋Δ • ίωΫγϣϯ͕ΔͷͰ࣮ࡍʹॲཧ͍ͯ͠ΔϦΫΤετͱണ͢Δɹ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 45
ryotarai/simproxy • h#ps:/ /github.com/ryotarai/simproxy • Go Ͱॻ͔ΕͨγϯϓϧͳϦόʔεϓϩΩγ • ϔϧενΣοΫͳͲͷجຊతͳػೳ͋Δ •
Balancing Method ʹ leastreq Λ࣮͍ͯ͠Δ • ࢦඪͱͯ͠୯७ͳίωΫγϣϯͰͳ࣍͘ͷΑ͏ͳΛར༻͢Δ • όοΫΤϯυ͕ड͚औ͕ͬͨԠ͕ฦ͖͍ͬͯͯͳ͍ϦΫΤετͷ • ͜ΕʹΑΓ HTTP keepalive ͕͋ͬͯ leastconn ͷΑ͏ͳόϥϯγϯά͕Մೳ • ϩʔυόϥϯγϯάͷΞϧΰϦζϜʹؔ͢Δ Systems Engineering ͱ࣮͢Δ So.ware Engineering hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 46
before simproxy hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy )
47
a"er simproxy hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy )
48
৽ଔݚम • ৽ଔʹରͯ͠ΠϯϑϥͷݚमΛ 3 ؒߦ͍ͬͯΔ • MySQL ͷΠϯσοΫεͷߏΛཧղ͠ΫΤϦνϡʔχϯάΛߦ͏ ... •
AWS ͷ ALB RDS ͳͲΛ͍ࣗͷΞϓϦέʔγϣϯΛεέʔϧͤ͞Δ ... • ࣮ࡍͷΞϓϦέʔγϣϯΛॻ͘ϝϯόʔ͕Πϯϑϥͷཧղ͕͋Δํ͕ྑ͍ • શͯͷΞϓϦέʔγϣϯίʔυΛ SRE ͕νΣοΫ͍ͯ͠Δͱεέʔϧ͠ͳ͍ • ྨ্ Overhead ͔͠Εͳ͍͕αʔϏεΛεέʔϧͤ͞ΔͨΊʹඞཁ • ࣮ࡍʹ Systems Engineering ͷਂ͍ݟ͕ແ͍ͱڭ͑Δ͜ͱ͕ग़དྷͳ͍ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 49
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 50
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 51
PR hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 52
We are hiring! • ΫοΫύουͰຊհͨ͠Α͏ͳऔΓΈΛߦ͍ͬͯ·͢ • ·ͩ·ͩղܾ͍ͨ͠ɾऔΓΈ͍ͨ՝͕ͨ͘͞Μ͋Δ • ָ͠Έͳ͕ΒҰॹʹΕΔϝϯόʔΛืू͍ͯ͠·͢ →
h$ps:/ /cookpad.jobs hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 53