Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ISUCON予選突破を支えるオペレーション技術 / Web Operations about ...
Search
Yuuki Tsubouchi (yuuk1)
August 21, 2016
Technology
1
23k
ISUCON予選突破を支えるオペレーション技術 / Web Operations about ISUCON
Kyoto.なんか#2の発表資料です。
ISUCON初出場のときに知りたかったことをまとめました。
技術だけでなく心構えのようなものも盛り込んでいます。
Yuuki Tsubouchi (yuuk1)
August 21, 2016
Tweet
Share
More Decks by Yuuki Tsubouchi (yuuk1)
See All by Yuuki Tsubouchi (yuuk1)
博士論文公聴会: Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining / PhD Defence
yuukit
0
19
博士学位論文予備審査 / Scaling Telemetry Workloads in Cloud Applications: Techniques for Instrumentation, Storage, and Mining
yuukit
1
1.7k
MetricSifter:クラウドアプリケーションにおける故障箇所特定の効率化のための多変量時系列データの特徴量削減 / FIT 2024
yuukit
2
190
工学としてのSRE再訪 / Revisiting SRE as Engineering
yuukit
19
13k
Cloudless Computingの論文紹介
yuukit
2
480
#SRE論文紹介 Detection is Better Than Cure: A Cloud Incidents Perspective V. Ganatra et. al., ESEC/FSE’23
yuukit
3
1.8k
エンジニアのためのSRE論文への招待 / Introduction to SRE Papers for Engineers
yuukit
2
11k
博士課程での研究まとめ 2023年1月版 / Summary of my research in the PhD course
yuukit
1
280
AI時代に向けたクラウドにおける信頼性エンジニアリングの未来構想 / DICOMO2022 6A-1
yuukit
7
3k
Other Decks in Technology
See All in Technology
OPENLOGI Company Profile
hr01
0
60k
Iceberg Meetup Japan #1 : Iceberg and Databricks
databricksjapan
0
300
大規模アジャイルフレームワークから学ぶエンジニアマネジメントの本質
staka121
PRO
3
410
速くて安いWebサイトを作る
nishiharatsubasa
15
15k
Reading Code Is Harder Than Writing It
trishagee
2
120
Raycast Favorites × Script Command で実現するお手軽情報チェック
smasato
1
130
クラウドサービス事業者におけるOSS
tagomoris
3
980
Windows の新しい管理者保護モード
murachiakira
0
200
次世代KYC活動報告 / 20250219-BizDay17-KYC-nextgen
oidfj
0
460
Goで作って学ぶWebSocket
ryuichi1208
3
2.5k
プロダクトエンジニア構想を立ち上げ、プロダクト志向な組織への成長を続けている話 / grow into a product-oriented organization
hiro_torii
1
340
データマネジメントのトレードオフに立ち向かう
ikkimiyazaki
6
1.2k
Featured
See All Featured
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
160
15k
It's Worth the Effort
3n
184
28k
Building a Scalable Design System with Sketch
lauravandoore
461
33k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
46
2.3k
The Invisible Side of Design
smashingmag
299
50k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
44
7k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Why You Should Never Use an ORM
jnunemaker
PRO
55
9.2k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Rebuilding a faster, lazier Slack
samanthasiow
80
8.8k
Put a Button on it: Removing Barriers to Going Fast.
kastner
60
3.7k
Transcript
ISUCON༧બಥഁΛࢧ͑Δ ΦϖϨʔγϣϯٕज़ id:y_uuki Kyoto.ͳΜ͔ #2
ISUCON ͓ͱͳΔWebαʔϏεΛܾΊΒΕͨϨΪϡϨʔγϣ ϯͷதͰݶք·ͰߴԽΛਤΔνϡʔχϯάότϧ
ISUCONॳग़ͷͱ͖ʹ ͕ࣗΓ͔ͨͬͨ͜ͱ
w/id:motemen id:Songmu ຊઓग़ => ࢂഊ
w/id:astj id:ntakanashi ༧બ શମ5Ґ ຊઓग़ => ࢂഊ
None
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
ISUCONͰΔ͜ͱ • ༩͑ΒΕͨΣϒΞϓϦέʔγϣϯΛͱʹ͔͘ߴԽ • ߴԽ • ϨεϙϯελΠϜΛখ͘͢͞Δ • εϧʔϓοτ (req/s)
͕େ͖͘͢Δ • CPUωοτϫʔΫͳͲϦιʔεফඅྔΛখ͘͢͞Δ • جຊతʹϨεϙϯελΠϜΛখ͘͢͞Δ͜ͱΛࢦ͢
ISUCONͰΔ͜ͱ • αʔόؚΉڥߏங • OSɾϛυϧΣΞͷνϡʔχϯά • ΞϓϦέʔγϣϯϩδοΫͱσʔλߏͷվળ Ops ※ඞͣ͠ҰਓͰશ෦Δඞཁͳ͍
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
ΦϖϨʔγϣϯͰΛकΓ ϓϩάϥϛϯάͰΛͱΔ
OpsकΓ • OSɾϛυϧΣΞͷνϡʔχϯά͕ܶతͳՃཁૉʹ ͳΔ͜ͱ͋·Γͳ͍ • OSɾϛυϧΣΞͷνϡʔχϯάϦιʔεফඅྔΛ খ͘͢͞Δ͜ͱ͕ଟ͍ • େ෯ͳݮཁૉΛͳ͘͢Πϝʔδ
Dev߈Ί • σʔλߏͷվળࠜຊྍ๏ • Ωϟογϡ/ΦϯϝϞϦຑༀ
Ops࢘ྩౝ • OpsγεςϜશମΛ၆ᛌ͢Δ • ঢ়گஅͱํܾఆ • ΞΫηεϩάαʔόͷෛՙঢ়گΛղੳ • ϘτϧωοΫಛఆ
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
ISUCONͰΔ͜ͱ • αʔόؚΉڥߏங • OSɾϛυϧΣΞͷνϡʔχϯά • ΞϓϦέʔγϣϯϩδοΫͱσʔλߏͷվળ Ops
ڥߏங • ΫϥυڥηοτΞοϓ (༧બͷ߹) • ISUCON4:AWS ISUCON5:GCE ISUCON6:Azure • ΞΧϯτ࡞ͱΠϯελϯε࡞
• αʔόϩάΠϯηοτΞοϓ • ϩάΠϯϢʔβ࡞ • ϝϯόʔશһͷsshͷެ։伴Λઃఆ • ͓ΞϓϦέʔγϣϯͷىಈ • ϛυϧΣΞͷΠϯετʔϧ • σϓϩΠͷࣗಈԽ
ϘτϧωοΫղੳ • nginxͰΞΫηεϩάΛग़ྗ => URL͝ͱʹղੳ • MySQLͰεϩʔΫΤϦϩά => pt-query-digestͰղੳ •
ϓϩϑΝΠϥͰίʔυϨϕϧͰ࣮ߦ࣌ؒղੳ • Perlͷ߹ Devel::NYTProf
Opsνϡʔχϯά • OS • MySQL • nginx • memcached •
redis • ΞϓϦέʔγϣϯͷWebαʔό·ΘΓ
ΞϓϦέʔγϣϯ֬ೝ • Opsʹͱͬͯ݁ߏݟམͱ͕ͪ͠ • ΞϓϦέʔγϣϯڍಈɺΞϓϦέʔγϣϯϩδοΫɺ σʔλߏ • ؾ͍ͮͨΒɺԿͷΞϓϦέʔγϣϯ͔Βͣʹ࡞ۀͯ͠ ͍ͨ •
MySQLͷίϯιʔϧʹೖͬͯɺ • ΞϓϦέʔγϣϯ୲͕֬ೝ͍ͯ͠ΔͣͳͷͰڭ͑ͯ Β͏͔ɺ࠷ॳʹΈΜͳͰ֬ೝ͢Δ
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
σϓϩΠͷࣗಈԽ • ඞͣεΫϦϓτΛ༻ҙ͢ΔɻࡶͳͷͰΑ͍ɻ • Slack௨ • git pull • carton
install (bundle install) • systemctl restart mysql • systemctl restart isuxi.perl • systemctl restart nginx
deploy.sh HOSTS=("$@") USERNAME=$USER for HOST in ${HOSTS[@]}; do ssh isucon@$HOST
"/home/isucon/notify.sh $USERNAME '$HOST deploying...' && cd /home/isucon/deploy && git pull && ~/deploy/env.sh carton install && sudo systemctl restart mysql && sudo service memcached restart && sudo systemctl restart isuxi.perl && sudo systemctl restart nginx && sudo sysctl -p && /home/isucon/notify.sh $USERNAME '$HOST deploy done'" done
ࢹ • ؾ͍ͮͨΒΞϓϦέʔγϣϯ͕མͪͯͯɺؒҧ͑ͯϕϯ νϚʔΫ࣮ߦ • ϕϯνϚʔΫΛ͙͢ʹ࣮ߦͰ͖ͳ͍߹͕͋Δ • MackerelͰ֎ܗࢹͯ͠མͪͯͳ͍͔νΣοΫ • Slack௨Ͱ͖ΔͷͰؾ͚ͮͯศར
ϛυϧΣΞͷΠϯετʔϧ • σϑΥϧτͰnginxɺMySQL͕Πϯετʔϧ͞Ε͍ͯΔ ͜ͱ͕ଟ͍ • ͏Մೳੑ͕ߴ͍ͷͰredismemcachedΛΠϯετʔ ϧ͢Δ • ιʔε͔ΒϏϧυͯ͠Α͍͕ɺόʔδϣϯʹͩ͜ΘΒ ͳ͍ͷͳΒɺapt-get
Ͱ͍ΕΕΑ͍
Ϧιʔε༻Ѳ • http://blog.yuuk.io/entry/linux-server-operations • ίΞ͝ͱͷCPUར༻ɺσΟεΫI/OɺϝϞϦ༻ྔ etc • ϘτϧωοΫಛఆͷώϯτʹͳΔ͜ͱ͕͋Δ
ΞΫηεϩάͷղੳ • proxy (eg.nginx) ͰΞΫηεϩάऔಘ • ϨεϙϯελΠϜϦΫΤετURL͕ͱΕΔ • alp (https://github.com/tkuchiki/alp)
ͳͲͰूܭ • ࣗ࡞εΫϦϓτ https://gist.github.com/yuuki/129983ab4b02e3a646ad • URL͝ͱͷɺϦΫΤετճɺ߹ܭ࣮ߦ࣌ؒɺฏۉ࣮ߦ࣌ؒͳͲग़ྗ isucon@isucon01:~$ sudo parse_axslog isucon5.access_log.tsv taken_sec req:GET / HTTP/1.1 taken_sum:474.08 req_count:714 avg_taken:0.66 req:GET /footprints HTTP/1.1 taken_sum:58.378 req_count:198 avg_taken:0.29 req:GET /friends HTTP/1.1 taken_sum:27.047 req_count:238 avg_taken:0.11 req:POST /diary/entry HTTP/1.1 taken_sum:6.51 req_count:195 avg_taken:0.03 …
MySQLεϩʔΫΤϦϩάͷղੳ • ͋ΔᮢҎ্ͷ࣮ߦ࣌ؒͷΫΤϦΛग़ྗ • long_query_time = 0 Ͱͯ͢ͷΫΤϦΛग़ྗ • pt-query-digestͰूܭ
# Profile # Rank Query ID Response time Calls R/Call V/M Item # ==== ================== ============= ===== ====== ===== =============== # 1 0x75B7218DD44BDBC5 27.6810 24.2% 63 0.4394 0.14 SELECT relations # 2 0x492C7CAB538DE80D 22.5882 19.7% 53 0.4262 0.25 SELECT footprints # 3 0xA44D49C56E635213 14.5630 12.7% 63 0.2312 0.02 SELECT entries # 4 0x57EBD74C3B600DB8 14.0403 12.3% 40 0.3510 0.02 SELECT entries # 5 0xDC67C8CE8419EE5B 12.8485 11.2% 7 1.8355 0.19 SELECT entries
OSͷνϡʔχϯά $ cat /etc/sysctl.d/99-sysctl.conf net.ipv4.tcp_max_tw_buckets = 2000000 net.ipv4.ip_local_port_range = 10000
65000 net.core.somaxconn = 32768 net.core.netdev_max_backlog = 8192 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 10 fs.file-max = 100000 Systemdͷ߹ɺόʔδϣϯ207ͱ21x͔Β/etc/sysctl.conf͕ ແޮ
nginxͷνϡʔχϯά worker_processes 1; worker_rlimit_nofile 65535; events { worker_connections 8192; }
http { include mime.types; sendfile on; keepalive_timeout 120s; tcp_nopush on; tcp_nodelay on; etag off;
MySQLͷνϡʔχϯά innodb_buffer_pool_size = 1G innodb_flush_log_at_trx_commit = 0 innodb_flush_method=O_DIRECT
PerlΞϓϦέʔγϣϯαʔόνϡʔχϯά start_server --backlog 16384 -- plackup -s Gazelle --max-reqs-per-child 50000
-workers=4 • ϫʔΧʔϓϩηείΞͱಉఔʹ͢Δ • ͜ΕҎ্૿ͯ͠ίϯςΩετεΠον૿͑Δ͚ͩ ͰείΞ৳ͼͳ͍͜ͱ͕ଟ͍ • preforkܕαʔόͰͳΔ͘fork͠ͳ͓ͤ͞ͳ͍Α͏ʹ
memcachedͷνϡʔχϯά -m 256 -U 0 -c 20000 -v -t 1
-C -B ascii
Redisͷνϡʔχϯά tcp-keepalive 60 save 900 1 save 300 10 save
120 20000 maxclients 10000 appendonly yes appendfsync everysec no-appendfsync-on-rewrite no
UnixυϝΠϯιέοτ • ༧બͰ1ͷαʔό্ʹɺ֤ϛυϧΣΞ͕ಉډ • proxy <=> app <=> db ͷଓ͕TCPͩͱ͍ͬͨͳ͍
• 3-way handshakeͦͷଞͷΦʔόϔου • ΦʔόϔουΛͳͨ͘͢ΊUnixυϝΠϯιέοτΛ͏ start_server --path /dev/shm/app.sock upstream app { server unix:/dev/shm/app.sock; } nginx <=> Plack ͷ߹
http://www.slideshare.net/kazeburo/isucon-summerclass2014action2final
͘͡ 1. ISUCONͰΔ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ
(࣮ફฤ) 5. ·ͱΊ
·ͱΊ • ISUCONͷΦϖϨʔγϣϯʹ͓͚Δయܕతͳܕͷ1ͭΛ հ • ISUCONͰϨεϙϯελΠϜΛͰ͖Δ͚ͩখ͘͢͞Δ • OpsकΓɺDev߈Ίͷݪଇ
Opsͷ͍͍ͱ͜Ζ • ൚༻తͳࣝΞϓϩʔνͰઓ͑Δ • શମΛΈͯΔײΛຯΘ͑Δ • ࣄલͷ࿅श͕Γ͍͢
ISUCON6༧બ 9݄17() ,18()