Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
モンストを支えるインフラの今とこれから
Isao Shimizu
March 01, 2016
Technology
71
12k
モンストを支えるインフラの今とこれから
dots. Conference Spring 2016
ゲーム開発の裏側
http://eventdots.jp/event/580344
Isao Shimizu
March 01, 2016
Tweet
Share
More Decks by Isao Shimizu
See All by Isao Shimizu
海外ユーザーのレスポンスタイム を可視化して改善した話 / Improved response times for users worldwide
isaoshimizu
1
66
約6年間運用したシステムをKubernetesに完全移行するまで/Kubernetes Novice Tokyo
isaoshimizu
6
1.4k
1,000万人以上が利用する「家族アルバム みてね」のSRE組織は4年間でどのように作られてきたのか/SRE NEXT 2022
isaoshimizu
7
9.6k
SREグループのマネージャーという立場になって真っ先にやったこと
isaoshimizu
5
1.7k
開発者とSREの役割、責任/SRE Lounge 13 LT
isaoshimizu
6
2.4k
世界中のユーザーが快適に利用できるクラウドネイティブなシステムを目指して/#CNDT2021
isaoshimizu
1
400
Ruby on Rails x Kubernetes におけるObservability / Rails x Kubernetes Observability
isaoshimizu
2
1.3k
利用者数800万人突破の「家族アルバム みてね」に学ぶ クラウドセキュリティの勘所/Cloud Security Tips
isaoshimizu
6
2.3k
「家族アルバム みてね」におけるAmazon EKSへの移行の取り組み / CNBF 202001
isaoshimizu
5
1.4k
Other Decks in Technology
See All in Technology
Deep dive in Reserved Instance ~脳死推奨量購入からの脱却~
kzkmaeda
0
550
OCI DevOps 概要 / OCI DevOps overview
oracle4engineer
PRO
0
510
OCI技術資料 : ロード・バランサー 詳細 / Load Balancer 200
ocise
2
7.2k
OVN-Kubernetes-Introduction-ja-2023-01-27.pdf
orimanabu
1
450
ユーザーテストガイドライン VERSION 2.0
kouzoukaikaku
0
1.5k
Logbii(ログビー) 会社紹介
logbii
0
170
OCIコンテナサービス関連の技術詳細 /oke-ocir-details
oracle4engineer
PRO
0
780
Exploring MapStore Release 2022.02: improved 3DTiles support and more
simboss
PRO
0
390
re:Inventで発表があったIoT事例の紹介と考察
kizawa2020
0
200
Multi-Cloud Gatewayでデータを統治せよ!/ Data Federation with MCG
tutsunom
1
350
Oracle Transaction Manager for Microservices Free 22.3 製品概要
oracle4engineer
PRO
5
110
re:Invent発表のサービスを取り入れて加速する弥生のSecurity&Governance / accelerating YAYOI's Security and Governance with services announced at reinvent
yayoi_dd
0
150
Featured
See All Featured
Building an army of robots
kneath
301
40k
From Idea to $5000 a Month in 5 Months
shpigford
374
44k
The Brand Is Dead. Long Live the Brand.
mthomps
48
2.9k
Debugging Ruby Performance
tmm1
67
11k
Three Pipe Problems
jasonvnalue
89
8.9k
Building a Modern Day E-commerce SEO Strategy
aleyda
6
4.5k
Building a Scalable Design System with Sketch
lauravandoore
451
31k
Designing for Performance
lara
600
65k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
182
15k
The Cult of Friendly URLs
andyhume
69
5.1k
Intergalactic Javascript Robots from Outer Space
tanoku
261
26k
A Modern Web Designer's Workflow
chriscoyier
689
180k
Transcript
ϞϯετΛࢧ͑ΔΠϯϑϥͷࠓͱ͜Ε͔Β 2016.3.1 dots. Conference Spring 2016 ήʔϜ։ൃͷཪଆ mixi, Inc. XFLAG™
STUDIO @isaoshimizu
About Me • ਗ਼ਫ ܄ SHIMIZU ISAO @isaoshimizu • 20118݄ΑΓגࣜձࣾϛΫγΟ
• ݱࡏΤοΫεϑϥοάελδΦ γεςϜ։ൃ෦ ॴଐ • 2લʹιʔγϟϧɾωοτϫʔΩϯά αʔϏε mixi ͷӡ༻͔Β εϚʔτϑΥϯήʔϜʮϞϯελʔετϥΠΫʯͷӡ༻ 2
ϞϯελʔετϥΠΫ • 201310݄10 ਖ਼ࣜϦϦʔε • iOS൛, Android൛Λఏڙ • શੈքͰ3,000ສਓҎ্͕ϓϨΠʢಉҰͰॏෳμϯϩʔυ͞Εؚͨ·ͣʣ •
ຊɺɺؖࠃɺถɺ߳ߓɾϚΧΦͰల։ • YouTubeͰΞχϝͷ৴ 2,500ສճͷ࠶ੜɺχϯςϯυʔ3DS൛ ग़ՙ100ສຊಥഁ 3
͔͜͜ΒϞϯελʔετϥΠΫ ຊ൛ʹݶఆͨ͠༰ʹͳΓ·͢ 4
ͰݟΔϞϯελʔετϥΠΫͷΠϯϑϥ 5
6 Application 300+ Servers Resque 50 Servers Redis 30 Servers
Memcached 200+ Servers CDN Traffic Max 270Gbps API Traffic Avg 1Gbps TURN 40 Servers MariaDB 250+ Servers Internal Traffic Max 20Gbps
Πϯϑϥߏ 7
8 DC1 DC2 Main Main/Backup Main/Dev
ෳDCͱAmazon VPC • DC 1ʢϝΠϯʣɺDC 2ʢϝΠϯͱόοΫΞοϓʣɺAWSʢϝΠϯͱ։ൃڥʣͰߏ • ֤DCTY2,4ܦ༝ͰAmazon VPCͱDirectConnectͰϓϥΠϕʔτଓ •
֤DCؒยܥো࣌ʹ͓͍ͯ20Gbps·Ͱ͑ΒΕΔઃܭ • ৄ͍͠ղઆ “ڌͨ͠ʢ෦τϥϑΟοΫϘϦϡʔϜฤʣ” http://xflag.com/blog/infradb/internal_traffic_volume.html 9
֤ڌͷׂ 10
11 DC1 DC2 Main Main/Backup Main/Dev Application MariaDB Redis Resque
Memcached MariaDB Redis Memcached TURN Server ։ൃ༻Πϯελϯε܈ Application Resque
֤ڌͷׂ • ຊ൪ͷApplicationɺResqueDC1ɺDC2ͷ྆ํͰՔಇ • ຊ൪ͷRedisɺMariaDBɺMemcached • DC1ʹMaster/Backupͷηοτ • DC2ʹBackup •
BackupDC1ͱDC2߹Θͤͯ2ηοτ͋Δ • AWSͰຊ൪ͷTURN ServerʢϚϧνϓϨΠ༻ʣɺ։ൃ༻Πϯελϯε܈ͳͲ͕Քಇɻ 12
ΫϥΠΞϯτΞϓϦ͔ΒݟͨΠϯϑϥ 13
14 DC1 DC2 AWS TURN Server (EC2) CDN (CloudFront, Akamai)
API API LB LB Internet Client
ΫϥΠΞϯτΞϓϦ͔ΒͷΞΫηεઌ • DC1ͱDC2ʹ͋ΔϩʔυόϥϯαʔʢHAߏʣ • ϩʔυόϥϯαʔͷΞΫηεDNSϥϯυϩϏϯʢ2ͭͷԾIPʣ • ϩʔυόϥϯαʔ͔Β֤ڌͷApplicationαʔόৼΓ͚ • AWSͰϚϧνϓϨΠ༻ʹTURN ServerʢEC2
+ Elastic IPʣ • ήʔϜσʔλɾϦιʔεσʔλ༻ʹCDNʢCloudFront + S3ʣɻAkamaiซ༻ɻ 15
σϓϩΠ 16
17 DC1 DC2 Deploy Server S3 Bucket tarball S3 PUT
GitHub ssh Code Internet
σϓϩΠ • Capistrano 2 • ඞཁͳͱ͖ʹඞཁͳ͚ͩσϓϩΠ͢Δ • capistrano-s3copy-awscli https://github.com/bacter1a/capistrano-s3copy-awscli •
GitHub͔Β࠷৽ͷίʔυΛऔಘͯ͠S3Ξοϓϩʔυ༻ʹtarballΛ࡞ • ApplicationαʔόͰS3ʹΞοϓϩʔυ͞ΕͨtarballΛμϯϩʔυɺల։ • GitHubͷଓ੍ݶରࡦʢgithub.comେྔʹଓ͕ཁٻ͞ΕΔͱڋ൱͞ΕΔͨΊʣ 18
• capistrano-rbenv https://github.com/yyuu/capistrano-rbenv • όʔδϣϯࢦఆͰRubyͷɺΞοϓσʔτָ͕ʹ • Ruby 2.2.4ಋೖࡁΈ 19
ϓϩϏδϣχϯά 20
21 DC1 DC2 Deploy Server S3 Bucket GitHub ssh Chef
Cookbook Ubuntu Package Mirror Internet deb Package deb Package
ϓϩϏδϣχϯά • ChefʹΑΔϓϩϏδϣχϯά͕جຊ • capistrano-paratrooper-chef https://github.com/tk0miya/capistrano-paratrooper-chef • ཧϚγϯͷॳظηοτΞοϓʢ࠷ݶʣAnsibleΛ͏ • OSΫϦʔϯΠϯετʔϧޙͷঢ়ଶͰɺChef͕Πϯετʔϧ͞Ε͍ͯͳ͍ͨΊ
22
• Community CookbookΛincludeͨ͠ಠࣗCookbookΛ࡞ɺར༻͍ͯ͠Δ • ಠࣗCookbookɺXFLAGશମͷΠϯϑϥͰڞ௨ͯ͑͠ΔΑ͏ʹ • ͯ͢ͷαʔό͕ChefͰߏஙՄೳ • Քಇதͷຊ൪DBɺMemcachedʹ࠶ద༻͕ͯ͠ى͖ͳ͍Α͏ʹ 23
• S3ʹMariaDBͷϨϙδτϦͷϛϥʔʢຊՈͷোରࡦʣ • ioDriveͳͲͷυϥΠόஔ͍ͯ͋Δ • aptlyͰߏங http://www.aptly.info/ • ݸਓͷ։ൃϚγϯͰVagrant +
VirtualBox + ChefͰߏͰ͖ΔΑ͏ʹ • Test Kitchenͷ׆༻ https://github.com/test-kitchen/test-kitchen 24
ෛՙରࡦ 25
ࠓʹೖ͔ͬͯΒߴෛՙ͕ଓ͘… 26
ෛՙରࡦ • DBͷγϟʔσΟϯάɺςʔϒϧׂΛܧଓ࣮ࢪ • Applicationαʔόͷ૿ઃ 12-20 core ͳCPU x 2ʢHyper-ThreadingࢉͰ24-40
coreʣͷϚγϯ͕جຊߏ • ΫΤϦվળΩϟογϡରԠ ioDrive(ioMemory)͔ΒSSDͷεέʔϧμϯͷ࣮ݱ • Memcachedͷ૿ઃ 27
Memcachedͷߏ 28
Memcachedͷߏ • Memcached2ϓʔϧߏ • 1ϓʔϧ͋ͨΓ100 • Ϛγϯ1͋ͨΓͷϝϞϦׂΓͯ26GBʢϝϞϦ32GBࡌͷϚγϯʣ • 1ϓʔϧશମͰ2.6TBͷ༰ྔΛͭ •
྆ํͷϓʔϧʹಉ࣌ॻ͖ࠐΈ DoubleWriteCacheStores https://github.com/hirocaster/double_write_cache_stores • ϝϯςφϯεແ͠ͰϓʔϧͷΓସ͕͑ՄೳɻϓʔϧαΠζͷ૿ڧ͕Մೳʹɻ 29
2ϓʔϧߏͷMemcached ೖΕସ͑ํ๏ 30
31 App M App App M M M M M
M M M M M M set get set Pool 1 Pool 2 App App App M M M M M M set get Pool 1 Pool 2 ௨ৗ࣌ ఀࢭɾ૿ઃ M M શΫϦΞ͓ͯ͘͠ M M M M M M Pool2ͷsetΛఀࢭ M Memcachedϗετ
32 App M App App M M M M M
set get set Pool 1 Pool 2 Γସ͑ M M M M M M M M App App App set get Pool 1 Pool 2 Ωϟογϡ͋ͨͨΊ set M M M M M M M M M M M M M M ͠Β͋ͨͨ͘ΊΔ
ϩʔυόϥϯαʔ 33
ϩʔυόϥϯαʔ • ApplicationαʔόͷલஈΞϓϥΠΞϯεͷϩʔυόϥϯαʔΛར༻ • A10 Networks AX2500 • RESTful APIʹΑΔૢ࡞͕Մೳ
• ࣗࣾ։ൃͷGoͷCLIπʔϧʢa10-cliʣ • ApplicationαʔόͷՃɺআ࣌ʹίϚϯυ1ͭͰαʔϏεΠϯɾΞτ͕Ͱ͖Δ 34
35 nginx unicorn nginx unicorn nginx unicorn nginx unicorn MemcachedMariaDBͳͲ
Internet A10 ϩʔυόϥϯαʔ Applicationαʔό
ϞχλϦϯάɾࢹ 36
37 CloudForecast • ͯ͢ͷαʔόͷϝτϦΫεऔಘ • 1,000ͷαʔό͕ܭଌର • ༷ʑͳಠࣗͷϝτϦΫεΛՃ https://github.com/kazeburo/cloudforecast
38 GrowthForecast • DCϥοΫͷిྗͷάϥϑԽ • Fluentd Monitoring AgentͷάϥϑԽ http://kazeburo.github.io/GrowthForecast/
39 • ͯ͢ͷαʔόͷࢹ • 12,000Λ͑Δࢹ߲ • ಠࣗͷϓϥάΠϯΛՃ • PagerDutyͱͷ࿈ܞ •
൪ͷεέδϡʔϦϯά • ΤεΧϨʔγϣϯϙϦγʔ • Slackͷ௨ https://www.nagios.org/
40 • nginx, ApplicationͷϩάΛ FluentdΛͬͯElasticsearchసૹ • APIຖͷॲཧ࣌ؒϨεϙϯελΠϜͷܭଌ • Τϥʔසͷܭଌ •
ಛఆϗετ͔ΒͷΞΫηεͷௐࠪ • Slow QueryͷՄࢹԽ • CloudTrailϩάͷՄࢹԽ https://www.elastic.co/
41 nginx unicorn Amazon S3 GrowthForecast LTSV Log Application Servers
ϩάసૹϑϩʔ Monitoring Agent Response Time Status Code Monitoring Agent Log Log td-agent 2.3.x Max 500Mbps Max 500Mbps
ৗͷπʔϧ 42
43 • ίϛϡχέʔγϣϯSlackͰ౷Ұ • Hubot https://hubot.github.com ར༻ʹΑΔChatOpsͷ࣮ݱ • ։ൃڥͷσϓϩΠͳͲΛSlack͔Β࣮ߦՄೳ •
ৄ͘͠Software Design 20161݄߸ͷChatOpsಛूʹ http://gihyo.jp/magazine/SD/archive/2016/201601 • ΞϓϦέʔγϣϯίʔυΠϯϑϥߏங༻ͷCookbookͳͲ ͯ͢GitHub্Ͱཧ • pull requestsϕʔεͷ։ൃɺϨϏϡʔ • GitHub্Ͱͷ׆ಈSlack௨͞ΕΔ • GitHubͰpull requests͕࡞͞ΕΔͱࣗಈςετ࣮ߦ • Dockerίϯςφͷੜɺςετ࣮ߦɺഁغ • Hubotͱͷ࿈ܞ
͜Ε͔Β͍͖͍ͬͯͨ͜ͱ 44
• ίʔυͷվળɺεέʔϧΞτɺΩϟογϡར༻ͳͲ • εέʔϧΞοϓͰ͖Δ͚ͩΓͨ͘ͳ͍ • ݁Ռతʹεέʔϧμϯʹͭͳ͕Δͱϕετ • ͜ΕΒैདྷ͔Βܧଓ࣮ͯ͠ࢪ͍ͯ͠Δ • ιϑτΣΞͷܧଓతΞοϓσʔτ
• Ubuntu Server 14.04 or 16.04 LTSʢ4/21ϦϦʔε༧ఆʣͷҠߦ • KernelɺϛυϧΣΞͷΞοϓσʔτੵۃతʹ࣮ࢪ 45
• MemcachedͷSlab Rebalancingʢ slab_reassign, slab_automove ʣ • ·ͩௐࠪɺݕূஈ֊ͳͷͰӡ༻ࣄྫ͕͋ΕΓ͍ͨ • ࢹɺϞχλϦϯάڥͷվળ
• ෛՙҟৗʹΑΓૉૣ͘ؾ͚ͮΔӡ༻ʹ • ͏গࣗ͠ಈԽ͍ͨ͠ͱ͜Ζ • Consulͱͷ࿈ܞ 46
• ࣾͷ৽نҊ݅ͰHashicorpࣾπʔϧ Terraformͷར༻͕࢝·͍ͬͯΔ • Ϟϯετʹޙ͔Βಋೖ͢ΔͷίετϦεΫ͕ߴ͍ͷͰಋೖ༧ఆͳ͠ • ෦తʹπʔϧར༻ʹΑΔޮԽ͓͜ͳ͍͖͍ͬͯͨ • ΠϯϑϥςετͷޮԽ •
DockerServerspecͳͲΛ͞Βʹ׆༻͍͖͍ͯͨ͠ • Chef CookbookͷϦϑΝΫλϦϯάɺͬͱγϯϓϧʹ 47
࠷ޙʹ 48
ΤϯδχΞืूதʂ 49 http://xflag.com/recruit/career/engineer.html
͋Γ͕ͱ͏͍͟͝·ͨ͠ 50