Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE at Cookpad
Search
Sponsored
·
SiteGround - Reliable hosting with speed, security, and support you can count on.
→
rrreeeyyy
November 29, 2017
Technology
1.4k
1
Share
SRE at Cookpad
#hbstudy の 79 回目で SRE の話をしました。
rrreeeyyy
November 29, 2017
More Decks by rrreeeyyy
See All by rrreeeyyy
Rethinking Incident Response: Context-Aware AI in Practice - Incident Buddy Edition -
rrreeeyyy
0
230
Rethinking Incident Response: Context-Aware AI in Practice
rrreeeyyy
3
2.4k
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
290
An Efficient Incident Response Training with AI / SRE NEXT 2024 Sponsor Session
rrreeeyyy
1
6k
カンファレンスから見る SRE トレンド 2024 / SRE Trends from Conferences in 2024 #SRE_Findy
rrreeeyyy
4
2.6k
信頼性の育て方 / mackerel-meetup-15
rrreeeyyy
10
2.9k
SRE の歩き方・進め方 / sre-walk-through-procedure
rrreeeyyy
0
9k
「信頼性」を保ちつつ大規模サービスをリニューアルする / cookpad-tech-kitchen-service-embedded-sres
rrreeeyyy
11
13k
Cookpad and Prometheus
rrreeeyyy
6
21k
Other Decks in Technology
See All in Technology
Practical TypeProf: Lessons from Analyzing Optcarrot
mame
0
330
#jawsugyokohama 100 LT11, "My AWS Journey 2011-2026 - kwntravel"
shinichirokawano
0
350
AIを共同作業者にして書籍を執筆する方法 / How to Write a Book with AI as a Co-Creator
ama_ch
2
130
AIはハッカーを減らすのか、増やすのか?──現役ホワイトハッカーから見るAI時代のリアル【MEGU-Meet】
cscengineer
0
140
AzureのIaC管理からログ調査まで、随所に役立つSkillsとCustom-Instructions / Boosting IaC and Log Analysis with Skills
aeonpeople
0
230
Rebirth of Software Craftsmanship in the AI Era
lemiorhan
PRO
4
2k
MLOps導入のための組織作りの第一歩
akasan
0
330
明日からドヤれる!超マニアックなAWSセキュリティTips10連発 / 10 Ultra-Niche AWS Security Tips
yuj1osm
0
590
AgentCore×VPCでの設計パターンn選と勘所
har1101
3
280
Do Vibe Coding ao LLM em Produção para Busca Agêntica - TDC 2026 - Summit IA - São Paulo
jpbonson
3
110
Standards et agents IA : un tour d’horizon de MCP, A2A, ADK et plus encore
glaforge
0
160
AI時代における技術的負債への取り組み
codenote
1
1.5k
Featured
See All Featured
The browser strikes back
jonoalderson
0
970
CSS Pre-Processors: Stylus, Less & Sass
bermonpainter
360
30k
The Curse of the Amulet
leimatthew05
1
11k
Marketing to machines
jonoalderson
1
5.2k
Jamie Indigo - Trashchat’s Guide to Black Boxes: Technical SEO Tactics for LLMs
techseoconnect
PRO
0
110
Future Trends and Review - Lecture 12 - Web Technologies (1019888BNR)
signer
PRO
0
3.5k
Visualization
eitanlees
150
17k
The Straight Up "How To Draw Better" Workshop
denniskardys
239
140k
Side Projects
sachag
455
43k
Building Applications with DynamoDB
mza
96
7k
Navigating Weather and Climate Data
rabernat
0
170
Become a Pro
speakerdeck
PRO
31
5.9k
Transcript
SRE at Cookpad ΫοΫύουגࣜձࣾ ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ ٢ ཽଠ (
@rrreeeyyy ) hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 1
Agenda • SRE ʹ͍ͭͯ(վΊͯ) • ΫοΫύουʹ͓͚ΔΠϯϑϥߏ • EC2 with spot
instances • hako + ECS ڥ • ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ • ࣄͷྲྀΕɾऔΓΈʹ͍ͭͯɹ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 2
!SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB3ZPUB Me • Yoshikawa Ryota ( @rrreeeyyy
[reɪ] ) • גࣜձࣾϋʔτϏʔπ (2010/11 ʙ 2016/12) • ΫοΫύουגࣜձࣾ (2017/01 ʙ) • ΠϯϑϥετϥΫνϟʔ෦ SRE άϧʔϓ • ڵຯྖҬ • ϞχλϦϯάɾ࣌ܥྻσʔλϕʔε • ࢄγεςϜɾϩʔυόϥϯα • झຯ • League of Legends, ΑΑΫϩχΫϧ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 3
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 4
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 5
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 6
(վΊͯ) Site Reliability Engineering • αΠτͷ৴པੑͷશͯʹൣғΛ࣋ͭ • ಛʹɺιϑτΣΞɾγεςϜΤϯδχΞϦϯάΛ༻͍ͯ՝Λղܾ͢Δ • ྫ͑ɺύϑΥʔϚϯε
• ύϑΥʔϚϯεʹ͕͋Δίʔυ SRE ͕ࣗίʔυΛॻ͍ͯमਖ਼͢Δ • ྫ͑ɺεέʔϥϏϦςΟɾΞϕΠϥϏϦςΟ • ਓؒʹΑΔखಈͷ࡞ۀΛίʔυʹΑΓࣗಈԽ͢Δ • ଐਓੑΛഉআͯ҆͠ఆͤ͞ΔɾਓؒΛ૿͞ͳͯ͘αʔϏεΛεέʔϧͤ͞Δ • ن͕૿େͯ҆͠ఆͯ͠αʔϏεΛఏڙͰ͖Δঢ়ଶʹ͢Δ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 7
SRE ͷ׆ಈ SRE ͷ׆ಈେ͖͚ͯ࣍͘ͷ 4 ͭʹྨͰ͖Δͱ͞Ε͍ͯΔ • So#ware Engineering •
Systems Engineering • Toil • Overhead hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 8
SRE ͷ׆ಈ • So#ware Engineering • ίʔυͷ࡞मਖ਼, ͦΕʹؔ࿈͢ΔઃܭυΩϡϝϯςʔγϣϯͷ࡞ۀͳͲ • ࣗಈԽεΫϦϓτͷ࡞,
πʔϧϑϨʔϜϫʔΫͷ࡞ ... • αʔϏεͷ৴པੑΛߴΊΔػೳͷՃ ... • Systems Engineering • 1 ճͷ࡞ۀͰӬଓతͳվળΛੜΉ࣍ͷΑ͏ͳ࡞ۀ • ຊ൪γεςϜͷઃఆมߋ ... • ։ൃνʔϜͷαʔϏεΞʔΩςΫνϟઃܭ ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 9
SRE ͷ׆ಈ • Toil • αʔϏεΛՔಇͤ͞ΔͨΊ͚ͩʹ܁Γฦ͠ߦΘΕΔखಈͰͷ࡞ۀ • ظతͳՁΛ࣋ͨͳ͍, αʔϏεͷʹରͯ͠ O(n)
Ͱ͋Δ ... • ͭ·Βͳ͍ࣄͰظతͳՁ͕͋Ε Toil Ͱͳ͍ • Overhead • αʔϏεΛՔಇͤ͞Δ͜ͱʹ݁͠ͳ͍ཧతͳ࡞ۀ • ࠾༻, ϛʔςΟϯά, ධՁ, τϨʔχϯά ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 10
ΫοΫύουʹ͓͚Δ SRE hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy )
11
ΫοΫύουʹ͓͚Δ SRE • ΠϯϑϥετϥΫνϟʔ෦ • ͷதʹ SRE άϧʔϓ͕͋Δ • DWH,
Security, ࣾIT ͕ಉ͡෦ॺʹ͋Δ • ϝϯόʔ SRE ͚ͩͰ 10 ਓఔ + ւ֎ʹ 2 ໊ఔ • ࠃ 7 ׂɾάϩʔόϧ 1 3 ׂ͙Β͍ͷൺ 1 h$ps:/ /speakerdeck.com/sorah/building-infrastructure-for-our-global-service hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 12
ΫοΫύουʹ͓͚ΔΠϯϑϥߏ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 13
ΫοΫύουʹ͓͚ΔΠϯϑϥߏ • ༷ʑ͋Δ͕ϝΠϯͰಈ͍͍ͯΔͷେ͖͚ͯ͘ 2 ͭ • EC2 + spotscaler ڥ
• EC2 ʹΞϓϦέʔγϣϯΛσϓϩΠ͍ͯ͠Δߏ • EC2 Πϯελϯεεϙοτʹͳ͍ͬͯΔ • ECS + hako ڥ • ECS Ϋϥελʹରͯ͠ίϯςφΛσϓϩΠ͍ͯ͠Δߏ • σϓϩΠϝϯτʹ eagletmt/hako Λར༻͍ͯ͠Δ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 14
EC2 + spotscaler ڥ • ΫοΫύουͷ͍ΘΏΔը໘ͷඳը෦ͳͲ͕͜Εʹ͋ͨΔ • PC ൛αΠτ, εϚϗ༻ͷ
API ... ͳͲ • Route53 + ALB + EC2 ͷ AWS ͷҰൠతͳߏ • EC2 spot instance Λ spotscaler 2 Λͬͯཧ͍ͯ͠Δ • deployment serf + sorah/mamiya 3 Λͬͯߦ͍ͬͯΔ 3 h$ps:/ /github.com/eagletmt/hako 2 h$ps:/ /speakerdeck.com/ryotarai/spot-instances-in-cookpad-number-cookpadtechconf-2017 hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 15
ECS + hako ڥ • ΫοΫύουͷ༷ʑͳϚΠΫϩαʔϏε͕͜Εʹ͋ͨΔ • λΠϜϥΠϯ, ࠂ, ೝূ...
ͳͲ • ػցֶशͷج൫͜͜ʹ͍ͬͯΔʢg2 Πϯελϯεͷ ECS Ϋϥελʣ • ECS ͷΫϥελʹରͯ͠ hako 3 Λར༻ͯ͠σϓϩΠ͍ͯ͠Δ • GHE ্ͷϦϙδτϦʹΞϓϦέʔγϣϯఆٛͷ PR Λग़ͯ͠Β͏ • σϓϩΠ Slack ্͔Β։ൃऀ͕جຊతʹࣗ༝ʹߦ͑Δ • PR ຖͷεςʔδϯάɾ౷߹ίϯιʔϧɾτϨʔγϯάͳͲ͕͋Δ 4 4 h$p:/ /techlife.cookpad.com/entry/2016/03/16/100043 3 h$ps:/ /github.com/eagletmt/hako hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 16
ECS + hako ڥ ʹ͓͚Δ YAML ઃఆ scheduler: <<: !include
schedulers/default.yml elb_v2: vpc_id: vpc-xxxxxx health_check_path: /healthcheck listeners: - port: 80 protocol: HTTP # : autoscaling: min_capacity: 2 max_capacity: 24 policies: # : policies for autoscaling hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 17
ECS + hako ڥ ʹ͓͚Δ YAML ઃఆ app: image: xxxxxxxxxx/service
# ECR Λࢦఆ͢Δ͜ͱ͕ଟ͍ cpu: 128 memory: 512 links: - fluentd env: $providers: - <<: !include env_providers/vault.yml directory: hako/group/service # vault hako/group/service ͷ - !include env_providers/vault_shared.yml # ֘ͷάϧʔϓʹॴଐ͢Δਓ͕ࣗ༝ʹઃఆͰ͖Δ RAILS_ENV: production WORKER_NUM: '2' additional_containers: front: !include containers/front.yml fluentd: !include containers/fluentd.yml volumes: td-agent: source_path: /var/lib/hako/td-agent/service hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 18
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 19
ΫοΫύουʹ͓͚ΔϚΠΫϩαʔϏεͷ࣮ફ • ৫ਓɾνʔϜɾαʔϏεͷʹͭΕͯݖݶͷҕৡ͕ඞཁʹͳͬͨ • SRE ج൫νʔϜ͕શͯΛཧ͢ΔͷෆՄೳͩ͠ෆ݈શ • ΞϓϦέʔγϣϯࣗମ͕େ͖͘ͳΓ༷͗ͯ͢ʑͳ͕ग़ΔΑ͏ʹͳͬͨ • ίʔυͷݟ௨͕͠ѱ͘ͳΔɾςετʹֻ͕͔࣌ؒΓ͗͢Δ
... • લ͔ΒঃʑʹඋΛਐΊ͍ͯΔ 4 4 h$p:/ /techlife.cookpad.com/entry/2016/03/16/100043 hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 20
ΫοΫύουʹ͓͚ΔϚΠΫϩαʔϏεͷ࣮ફ • جຊతʹϝΠϯͷΞϓϦέʔγϣϯΛ hako ڥʹΓग़͢ • ৽نʹ࡞͞ΕΔΞϓϦέʔγϣϯ hako ڥΛඪ४ʹ͢Δ •
ಉ࣌ʹɺmicroservice ԽʹΑΔ՝ʹ͍͔ͭ͘औΓΜͰ͍Δ • αʔϏεؒͷ Tracing ʹ AWS X-Ray Λར༻͍ͯ͠Δ • αʔϏεϝογϡͷ࣮ݱʹ envoyproxy/envoy Λར༻͍ͯ͠Δ • ͍͔ͭ͘՝͕͋Γ࣮ͭͭࡍʹಈ͔ͯ͠ΛਐΊ͍ͯΔ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 21
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 22
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 23
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy
) 24
ΫοΫύουʹ͓͚Δ SRE ͷ׆ಈ • ΫοΫύουͷ SRE ͷ׆ಈΛ͍͔ͭ͘հ • αʔόߏͷཧɾมߋ •
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • ϞχλϦϯάͷվળ • ϩʔυόϥϯγϯάͷվળ • ৽ଔݚम • ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 25
αʔόߏͷมߋɾཧ • αʔόߏجຊతʹશͯίʔυԽ͞Ε͍ͯΔ • Codenize.tools 5 Itamae 6, Terraform
ͳͲ • IAM User Route53 GHE ্Ͱී௨ʹཧ͞Ε͍ͯΔ • ։ൃऀ͕ࣗ༝ʹ PR Λग़͢͜ͱ͕ग़དྷΔ • αʔόߏͷมߋίʔυͷ Pull request ͱͯ͠ग़͢ • ϨϏϡʔΛड͚ɺϚʔδ͔ͯ͠Βө • Ұ෦Ϛʔδͯࣗ͠ಈͰө͞ΕΔ 6 h$ps:/ /github.com/itamae-kitchen/itamae 5 h$ps:/ /codenize.tools/ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 26
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 27
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy
) 28
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 29
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • େ͖ΊͳػೳαʔϏε୯ҐͰׂͯ͠։ൃ͢Δ • ߹ʹΑΔ͕ SRE ͕ 1
ਓΞαΠϯ͞ΕΔ • ֘ͷαʔϏεͷ৴པੑʹؔ͢Δ͜ͱجຊతʹશ෦Δ • ઃܭͷ૬ஊɾϨϏϡʔ • ϦϦʔεલͷෛՙࢼݧɾΩϟύγςΟϓϥϯχϯά • ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν • ... hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 30
ઃܭͷ૬ஊɾϨϏϡʔ • ։ൃͷલʹҎԼͷΑ͏ͳ͜ͱΛڞ༗ͯ͠Β͍ٞ͢Δ • ͲͷΑ͏ͳΞϓϦέʔγϣϯΛ࡞Δ͔ • ͲͷΑ͏ͳίϯϙʔωϯτ͕ඞཁ͔ • ͲͷΑ͏ͳߏɾεέʔϧઓུʹ͢Δ͔ •
Design Document Λॻ͍ͯΒ͍ͦΕΛϕʔεʹΛͨ͠Γ͢Δ • େ͖ΊͷػೳͳͲΛ։ൃ͢Δ࣌ Design Document Λॻ͘͜ͱ͕ଟ͍ (SRE ) hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 31
ෛՙࢼݧ • ͋Δఔग़དྷ͖ͯͨลΓͰෛՙࢼݧͷΛ͢Δ • Ͳͷ͘Β͍ͷϦΫΤετ͕ఆ͞ΕΔ͔ • ͲͷΑ͏ͳϢʔβϦΫΤετ͕ఆ͞ΕΔ͔ • SRE ଆͰෛՙࢼݧͷγφϦΦڥΛ࡞Δ
• ෛՙࢼݧΛߦ͍ͳ͕Βຊ൪ڥͰͷΩϟύγςΟΛݟੵΔ • ΞϓϦέʔγϣϯ্ͷϘτϧωοΫ͕Ͳ͜ʹ͋Δ͔ΛݟΔ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 32
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 33
ύϑΥʔϚϯεʹର͢ΔΞϓϩʔν • ৴པੑʹؔ͢Δ͕͋Δ߹ࣗͰίʔυΛॻ͘ • ྫ͑ύϑΥʔϚϯε্ͷͳͲ • ΞϓϦέʔγϣϯͷ༷Λཧղͯ͠దͳΞϓϩʔνΛऔΔ • ྫ͑Ͳ͏͍͏ը໘ͷͲ͏͍͏ΫΤϦͳͷ͔ •
Ͳ͏͍͏ใ͕ཉͯ͘͠Ͳ͏͍͏੍͕͋Δͷ͔ • ͍ͬͯΔϛυϧΣΞͷ༷ཧղͯ͠దͳΞϓϩʔνΛऔΔ • ྫ͑ SQL ͷվળͳΒ MySQL ͷΠϯσοΫεͷߏ෦ॲཧͷ͕ࣝ͋Δͱྑ͍ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 34
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 35
৽نαʔϏεʹର͢Δ SRE ͷಈ͖ • ͜ͷลΓͨΓલͷࣄΛී௨ʹ͍ͬͯΔ͚ͩ • ͬͱվળग़དྷΔՕॴ͕ͨ͘͞Μ͋Δͱߟ͍͑ͯΔ • ྫ͑ෛՙࢼݧΛࣗಈͰߦ͏Α͏ʹ͢ΔɺͳͲ •
ΞϓϦέʔγϣϯΛཧղ͠ɺඞཁͳΒίʔυΛॻ͘ • SRE αΠτͷ৴པੑʹؔΘΔશͯʹΛෛ͏ • ΞϓϦέʔγϣϯͷಛੑઃܭΛཧղ͠ϛυϧΣΞΛΈ߹ΘͤΔ Systems Engineering • ༷ϛυϧΣΞΛཧղ࣮͠ࡍʹΞϓϦέʔγϣϯͷϩδοΫΛॻ͘ So.ware Engineering hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 36
ϞχλϦϯάͷվળ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 37
ϞχλϦϯάͷվળ • ݱࡏϝτϦΫεऔಘʹ Zabbix Λ༻͍ͯ͠Δ • ΑΓεέʔϧ͠ɺߴڃݴޠͰϝτϦΫεΞϥʔτΛѻ͑ΔΑ͏ʹ͍ͨ͠ • SRE ຊ
ͷ 10 ষʹॻ͔Ε͍ͯΔΑ͏ͳঢ়ଶʹͳΓ͍ͨ • Prometheus ͷಋೖΛঃʑʹ࣮ࢪ͍ͯ͠Δ • ϝτϦΫεͷղ૾ 15 ඵ୯ҐͳͲͰऔಘ͢ΔΑ͏ʹ͍ͯ͠Δ • ୯७ͳ HTTP ϨεϙϯεͰϝτϦΫεΛऩूग़དྷΔ • ΞϓϦέʔγϣϯࣗମͷϝτϦΫεͳͲ͕ൺֱత؆୯ʹऩूग़དྷΔ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 38
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 39
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 40
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 41
ϞχλϦϯάͷվળ • Ͳ͏͍ͬͨϛυϧΣΞΛબఆ͢Δ͔ • ϞχλϦϯάͰ͋Ε࣌ܥྻσʔλϕʔεͳͲʹؔ͢Δਂ͍ཧղΛ͔ͯ͠ΒબͿ • Ͳ͏͍ͬͨಛੑ͕͋Δ͔ʁࣗͨͪͷنʹ߹͍ͬͯΔ͔ʁεέʔϧ͢Δ͔ʁ • Γͳ͍ίϯϙʔωϯτͳ͍͔ʁແ͍ͳΒࣗ࡞Ͱ͖ͦ͏ͳن͔ʁ •
Prometheus ͳΒྫ͑ Long-Term Storage पΓͷίϯϙʔωϯτͳͲ • ͔ͬ͠Γͱͨ͠ Systems Engineering ͷݟΛར༻ͯ͠બఆ͢Δ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 42
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 43
ϩʔυόϥϯγϯάͷվળ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 44
ϩʔυόϥϯγϯάͷվળ • ैདྷΞϓϦέʔγϣϯαʔόͷϩʔυόϥϯγϯάʹ haproxy Λར༻͍ͯͨ͠ • Weighted Round-robin (wrr) Λར༻ͯ͠όϥϯγϯάΛߦ͍ͬͯͨ
• ෛՙͷ͍ΤϯυϙΠϯτͱߴ͍ΤϯυϙΠϯτ͕͋Δ߹ෛՙ͕Βͭ͘ • wrr ͩͱಛఆͷαʔόʹෛՙͷߴ͍ΤϯυϙΠϯτ͕ूத͢ΔՄೳੑ͕͋Δ • leastconn Λ͑ྑͦ͞͏͕ͩ HTTP keepalive ͍ͯ͠Δ߹ʹ͕͋Δ • ίωΫγϣϯ͕ΔͷͰ࣮ࡍʹॲཧ͍ͯ͠ΔϦΫΤετͱണ͢Δɹ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 45
ryotarai/simproxy • h#ps:/ /github.com/ryotarai/simproxy • Go Ͱॻ͔ΕͨγϯϓϧͳϦόʔεϓϩΩγ • ϔϧενΣοΫͳͲͷجຊతͳػೳ͋Δ •
Balancing Method ʹ leastreq Λ࣮͍ͯ͠Δ • ࢦඪͱͯ͠୯७ͳίωΫγϣϯͰͳ࣍͘ͷΑ͏ͳΛར༻͢Δ • όοΫΤϯυ͕ड͚औ͕ͬͨԠ͕ฦ͖͍ͬͯͯͳ͍ϦΫΤετͷ • ͜ΕʹΑΓ HTTP keepalive ͕͋ͬͯ leastconn ͷΑ͏ͳόϥϯγϯά͕Մೳ • ϩʔυόϥϯγϯάͷΞϧΰϦζϜʹؔ͢Δ Systems Engineering ͱ࣮͢Δ So.ware Engineering hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 46
before simproxy hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy )
47
a"er simproxy hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy )
48
৽ଔݚम • ৽ଔʹରͯ͠ΠϯϑϥͷݚमΛ 3 ؒߦ͍ͬͯΔ • MySQL ͷΠϯσοΫεͷߏΛཧղ͠ΫΤϦνϡʔχϯάΛߦ͏ ... •
AWS ͷ ALB RDS ͳͲΛ͍ࣗͷΞϓϦέʔγϣϯΛεέʔϧͤ͞Δ ... • ࣮ࡍͷΞϓϦέʔγϣϯΛॻ͘ϝϯόʔ͕Πϯϑϥͷཧղ͕͋Δํ͕ྑ͍ • શͯͷΞϓϦέʔγϣϯίʔυΛ SRE ͕νΣοΫ͍ͯ͠Δͱεέʔϧ͠ͳ͍ • ྨ্ Overhead ͔͠Εͳ͍͕αʔϏεΛεέʔϧͤ͞ΔͨΊʹඞཁ • ࣮ࡍʹ Systems Engineering ͷਂ͍ݟ͕ແ͍ͱڭ͑Δ͜ͱ͕ग़དྷͳ͍ hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 49
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 50
hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 51
PR hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 52
We are hiring! • ΫοΫύουͰຊհͨ͠Α͏ͳऔΓΈΛߦ͍ͬͯ·͢ • ·ͩ·ͩղܾ͍ͨ͠ɾऔΓΈ͍ͨ՝͕ͨ͘͞Μ͋Δ • ָ͠Έͳ͕ΒҰॹʹΕΔϝϯόʔΛืू͍ͯ͠·͢ →
h$ps:/ /cookpad.jobs hbstudy#79 (2017/11/20) | Yoshikawa Ryota ( @rrreeeyyy ) 53