Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ISUCON予選突破を支えるオペレーション技術 / Web Operations about ...
Search
Yuuki Tsubouchi (yuuk1)
August 21, 2016
Technology
1
23k
ISUCON予選突破を支えるオペレーション技術 / Web Operations about ISUCON
Kyoto.なんか#2の発表資料です。
ISUCON初出場のときに知りたかったことをまとめました。
技術だけでなく心構えのようなものも盛り込んでいます。
Yuuki Tsubouchi (yuuk1)
August 21, 2016
Tweet
Share
More Decks by Yuuki Tsubouchi (yuuk1)
See All by Yuuki Tsubouchi (yuuk1)
博士論文公聴会: Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining / PhD Defence
yuukit
1
25
博士学位論文予備審査 / Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining
yuukit
1
1.7k
MetricSifter:クラウドアプリケーションにおける故障箇所特定の効率化のための多変量時系列データの特徴量削減 / FIT 2024
yuukit
2
190
工学としてのSRE再訪 / Revisiting SRE as Engineering
yuukit
19
13k
Cloudless Computingの論文紹介
yuukit
2
490
#SRE論文紹介 Detection is Better Than Cure: A Cloud Incidents Perspective V. Ganatra et. al., ESEC/FSE’23
yuukit
3
1.8k
エンジニアのためのSRE論文への招待 / Introduction to SRE Papers for Engineers
yuukit
2
11k
博士課程での研究まとめ 2023年1月版 / Summary of my research in the PhD course
yuukit
1
280
AI時代に向けたクラウドにおける信頼性エンジニアリングの未来構想 / DICOMO2022 6A-1
yuukit
7
3k
Other Decks in Technology
See All in Technology
PHPで印刷所に入稿できる名札データを作る / Generating Print-Ready Name Tag Data with PHP
tomzoh
0
180
役員・マネージャー・著者・エンジニアそれぞれの立場から見たAWS認定資格
nrinetcom
PRO
3
6k
偏光画像処理ライブラリを作った話
elerac
1
170
Perlの生きのこり - エンジニアがこの先生きのこるためのカンファレンス2025
kfly8
2
270
【内製開発Summit 2025】イオンスマートテクノロジーの内製化組織の作り方/In-house-development-summit-AST
aeonpeople
2
660
システム・ML活用を広げるdbtのデータモデリング / Expanding System & ML Use with dbt Modeling
i125
1
320
OCI Success Journey OCIの何が評価されてる?疑問に答える事例セミナー(2025年2月実施)
oracle4engineer
PRO
2
150
内製化を加速させるlaC活用術
nrinetcom
PRO
2
140
遷移の高速化 ヤフートップの試行錯誤
narirou
6
1.2k
ExaDB-XSで利用されているExadata Exascaleについて
oracle4engineer
PRO
3
250
Snowflakeの開発・運用コストをApache Icebergで効率化しよう!~機能と活用例のご紹介~
sagara
1
450
LINEギフトにおけるバックエンド開発
lycorptech_jp
PRO
0
280
Featured
See All Featured
Site-Speed That Sticks
csswizardry
4
410
Six Lessons from altMBA
skipperchong
27
3.6k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
29
1k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.9k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
226
22k
Docker and Python
trallard
44
3.3k
Building Your Own Lightsaber
phodgson
104
6.2k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
49k
Practical Orchestrator
shlominoach
186
10k
How to Ace a Technical Interview
jacobian
276
23k
Transcript
ISUCON༧બಥഁΛࢧ͑Δ ΦϖϨʔγϣϯٕज़ id:y_uuki Kyoto.ͳΜ͔ #2
ISUCON ͓ͱͳΔWebαʔϏεΛܾΊΒΕͨϨΪϡϨʔγϣ ϯͷதͰݶք·ͰߴԽΛਤΔνϡʔχϯάότϧ
ISUCONॳग़ͷͱ͖ʹ ͕ࣗΓ͔ͨͬͨ͜ͱ
w/id:motemen id:Songmu ຊઓग़ => ࢂഊ
w/id:astj id:ntakanashi ༧બ શମ5Ґ ຊઓग़ => ࢂഊ
None
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
ISUCONͰΔ͜ͱ • ༩͑ΒΕͨΣϒΞϓϦέʔγϣϯΛͱʹ͔͘ߴԽ • ߴԽ • ϨεϙϯελΠϜΛখ͘͢͞Δ • εϧʔϓοτ (req/s)
͕େ͖͘͢Δ • CPUωοτϫʔΫͳͲϦιʔεফඅྔΛখ͘͢͞Δ • جຊతʹϨεϙϯελΠϜΛখ͘͢͞Δ͜ͱΛࢦ͢
ISUCONͰΔ͜ͱ • αʔόؚΉڥߏங • OSɾϛυϧΣΞͷνϡʔχϯά • ΞϓϦέʔγϣϯϩδοΫͱσʔλߏͷվળ Ops ※ඞͣ͠ҰਓͰશ෦Δඞཁͳ͍
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
ΦϖϨʔγϣϯͰΛकΓ ϓϩάϥϛϯάͰΛͱΔ
OpsकΓ • OSɾϛυϧΣΞͷνϡʔχϯά͕ܶతͳՃཁૉʹ ͳΔ͜ͱ͋·Γͳ͍ • OSɾϛυϧΣΞͷνϡʔχϯάϦιʔεফඅྔΛ খ͘͢͞Δ͜ͱ͕ଟ͍ • େ෯ͳݮཁૉΛͳ͘͢Πϝʔδ
Dev߈Ί • σʔλߏͷվળࠜຊྍ๏ • Ωϟογϡ/ΦϯϝϞϦຑༀ
Ops࢘ྩౝ • OpsγεςϜશମΛ၆ᛌ͢Δ • ঢ়گஅͱํܾఆ • ΞΫηεϩάαʔόͷෛՙঢ়گΛղੳ • ϘτϧωοΫಛఆ
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
ISUCONͰΔ͜ͱ • αʔόؚΉڥߏங • OSɾϛυϧΣΞͷνϡʔχϯά • ΞϓϦέʔγϣϯϩδοΫͱσʔλߏͷվળ Ops
ڥߏங • ΫϥυڥηοτΞοϓ (༧બͷ߹) • ISUCON4:AWS ISUCON5:GCE ISUCON6:Azure • ΞΧϯτ࡞ͱΠϯελϯε࡞
• αʔόϩάΠϯηοτΞοϓ • ϩάΠϯϢʔβ࡞ • ϝϯόʔશһͷsshͷެ։伴Λઃఆ • ͓ΞϓϦέʔγϣϯͷىಈ • ϛυϧΣΞͷΠϯετʔϧ • σϓϩΠͷࣗಈԽ
ϘτϧωοΫղੳ • nginxͰΞΫηεϩάΛग़ྗ => URL͝ͱʹղੳ • MySQLͰεϩʔΫΤϦϩά => pt-query-digestͰղੳ •
ϓϩϑΝΠϥͰίʔυϨϕϧͰ࣮ߦ࣌ؒղੳ • Perlͷ߹ Devel::NYTProf
Opsνϡʔχϯά • OS • MySQL • nginx • memcached •
redis • ΞϓϦέʔγϣϯͷWebαʔό·ΘΓ
ΞϓϦέʔγϣϯ֬ೝ • Opsʹͱͬͯ݁ߏݟམͱ͕ͪ͠ • ΞϓϦέʔγϣϯڍಈɺΞϓϦέʔγϣϯϩδοΫɺ σʔλߏ • ؾ͍ͮͨΒɺԿͷΞϓϦέʔγϣϯ͔Βͣʹ࡞ۀͯ͠ ͍ͨ •
MySQLͷίϯιʔϧʹೖͬͯɺ • ΞϓϦέʔγϣϯ୲͕֬ೝ͍ͯ͠ΔͣͳͷͰڭ͑ͯ Β͏͔ɺ࠷ॳʹΈΜͳͰ֬ೝ͢Δ
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
σϓϩΠͷࣗಈԽ • ඞͣεΫϦϓτΛ༻ҙ͢ΔɻࡶͳͷͰΑ͍ɻ • Slack௨ • git pull • carton
install (bundle install) • systemctl restart mysql • systemctl restart isuxi.perl • systemctl restart nginx
deploy.sh HOSTS=("$@") USERNAME=$USER for HOST in ${HOSTS[@]}; do ssh isucon@$HOST
"/home/isucon/notify.sh $USERNAME '$HOST deploying...' && cd /home/isucon/deploy && git pull && ~/deploy/env.sh carton install && sudo systemctl restart mysql && sudo service memcached restart && sudo systemctl restart isuxi.perl && sudo systemctl restart nginx && sudo sysctl -p && /home/isucon/notify.sh $USERNAME '$HOST deploy done'" done
ࢹ • ؾ͍ͮͨΒΞϓϦέʔγϣϯ͕མͪͯͯɺؒҧ͑ͯϕϯ νϚʔΫ࣮ߦ • ϕϯνϚʔΫΛ͙͢ʹ࣮ߦͰ͖ͳ͍߹͕͋Δ • MackerelͰ֎ܗࢹͯ͠མͪͯͳ͍͔νΣοΫ • Slack௨Ͱ͖ΔͷͰؾ͚ͮͯศར
ϛυϧΣΞͷΠϯετʔϧ • σϑΥϧτͰnginxɺMySQL͕Πϯετʔϧ͞Ε͍ͯΔ ͜ͱ͕ଟ͍ • ͏Մೳੑ͕ߴ͍ͷͰredismemcachedΛΠϯετʔ ϧ͢Δ • ιʔε͔ΒϏϧυͯ͠Α͍͕ɺόʔδϣϯʹͩ͜ΘΒ ͳ͍ͷͳΒɺapt-get
Ͱ͍ΕΕΑ͍
Ϧιʔε༻Ѳ • http://blog.yuuk.io/entry/linux-server-operations • ίΞ͝ͱͷCPUར༻ɺσΟεΫI/OɺϝϞϦ༻ྔ etc • ϘτϧωοΫಛఆͷώϯτʹͳΔ͜ͱ͕͋Δ
ΞΫηεϩάͷղੳ • proxy (eg.nginx) ͰΞΫηεϩάऔಘ • ϨεϙϯελΠϜϦΫΤετURL͕ͱΕΔ • alp (https://github.com/tkuchiki/alp)
ͳͲͰूܭ • ࣗ࡞εΫϦϓτ https://gist.github.com/yuuki/129983ab4b02e3a646ad • URL͝ͱͷɺϦΫΤετճɺ߹ܭ࣮ߦ࣌ؒɺฏۉ࣮ߦ࣌ؒͳͲग़ྗ isucon@isucon01:~$ sudo parse_axslog isucon5.access_log.tsv taken_sec req:GET / HTTP/1.1 taken_sum:474.08 req_count:714 avg_taken:0.66 req:GET /footprints HTTP/1.1 taken_sum:58.378 req_count:198 avg_taken:0.29 req:GET /friends HTTP/1.1 taken_sum:27.047 req_count:238 avg_taken:0.11 req:POST /diary/entry HTTP/1.1 taken_sum:6.51 req_count:195 avg_taken:0.03 …
MySQLεϩʔΫΤϦϩάͷղੳ • ͋ΔᮢҎ্ͷ࣮ߦ࣌ؒͷΫΤϦΛग़ྗ • long_query_time = 0 Ͱͯ͢ͷΫΤϦΛग़ྗ • pt-query-digestͰूܭ
# Profile # Rank Query ID Response time Calls R/Call V/M Item # ==== ================== ============= ===== ====== ===== =============== # 1 0x75B7218DD44BDBC5 27.6810 24.2% 63 0.4394 0.14 SELECT relations # 2 0x492C7CAB538DE80D 22.5882 19.7% 53 0.4262 0.25 SELECT footprints # 3 0xA44D49C56E635213 14.5630 12.7% 63 0.2312 0.02 SELECT entries # 4 0x57EBD74C3B600DB8 14.0403 12.3% 40 0.3510 0.02 SELECT entries # 5 0xDC67C8CE8419EE5B 12.8485 11.2% 7 1.8355 0.19 SELECT entries
OSͷνϡʔχϯά $ cat /etc/sysctl.d/99-sysctl.conf net.ipv4.tcp_max_tw_buckets = 2000000 net.ipv4.ip_local_port_range = 10000
65000 net.core.somaxconn = 32768 net.core.netdev_max_backlog = 8192 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 10 fs.file-max = 100000 Systemdͷ߹ɺόʔδϣϯ207ͱ21x͔Β/etc/sysctl.conf͕ ແޮ
nginxͷνϡʔχϯά worker_processes 1; worker_rlimit_nofile 65535; events { worker_connections 8192; }
http { include mime.types; sendfile on; keepalive_timeout 120s; tcp_nopush on; tcp_nodelay on; etag off;
MySQLͷνϡʔχϯά innodb_buffer_pool_size = 1G innodb_flush_log_at_trx_commit = 0 innodb_flush_method=O_DIRECT
PerlΞϓϦέʔγϣϯαʔόνϡʔχϯά start_server --backlog 16384 -- plackup -s Gazelle --max-reqs-per-child 50000
-workers=4 • ϫʔΧʔϓϩηείΞͱಉఔʹ͢Δ • ͜ΕҎ্૿ͯ͠ίϯςΩετεΠον૿͑Δ͚ͩ ͰείΞ৳ͼͳ͍͜ͱ͕ଟ͍ • preforkܕαʔόͰͳΔ͘fork͠ͳ͓ͤ͞ͳ͍Α͏ʹ
memcachedͷνϡʔχϯά -m 256 -U 0 -c 20000 -v -t 1
-C -B ascii
Redisͷνϡʔχϯά tcp-keepalive 60 save 900 1 save 300 10 save
120 20000 maxclients 10000 appendonly yes appendfsync everysec no-appendfsync-on-rewrite no
UnixυϝΠϯιέοτ • ༧બͰ1ͷαʔό্ʹɺ֤ϛυϧΣΞ͕ಉډ • proxy <=> app <=> db ͷଓ͕TCPͩͱ͍ͬͨͳ͍
• 3-way handshakeͦͷଞͷΦʔόϔου • ΦʔόϔουΛͳͨ͘͢ΊUnixυϝΠϯιέοτΛ͏ start_server --path /dev/shm/app.sock upstream app { server unix:/dev/shm/app.sock; } nginx <=> Plack ͷ߹
http://www.slideshare.net/kazeburo/isucon-summerclass2014action2final
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
·ͱΊ • ISUCONͷΦϖϨʔγϣϯʹ͓͚Δయܕతͳܕͷ1ͭΛ հ • ISUCONͰϨεϙϯελΠϜΛͰ͖Δ͚ͩখ͘͢͞Δ • OpsकΓɺDev߈Ίͷݪଇ
Opsͷ͍͍ͱ͜Ζ • ൚༻తͳࣝΞϓϩʔνͰઓ͑Δ • શମΛΈͯΔײΛຯΘ͑Δ • ࣄલͷ࿅श͕Γ͍͢
ISUCON6༧બ 9݄17() ,18()