ISUCON予選突破を支えるオペレーション技術 / Web Operations about ISUCON

Kyoto.なんか#2の発表資料です。
ISUCON初出場のときに知りたかったことをまとめました。
技術だけでなく心構えのようなものも盛り込んでいます。

Transcript

1. ISUCON༧બಥഁΛࢧ͑Δ ΦϖϨʔγϣϯٕज़ id:y_uuki Kyoto.ͳΜ͔ #2
2. ISUCON ͓୊ͱͳΔWebαʔϏεΛܾΊΒΕͨϨΪϡϨʔγϣ ϯͷதͰݶք·Ͱߴ଎ԽΛਤΔνϡʔχϯάότϧ
3. ISUCONॳग़৔ͷͱ͖ʹ ࣗ෼͕஌Γ͔ͨͬͨ͜ͱ
4. w/id:motemen id:Songmu ຊઓग़৔ => ࢂഊ
5. w/id:astj id:ntakanashi ༧બ શମ5Ґ ຊઓग़৔ => ࢂഊ
7. ΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ
8. ΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ
9. ISUCONͰ΍Δ͜ͱ • ༩͑ΒΕͨ΢ΣϒΞϓϦέʔγϣϯΛͱʹ͔͘ߴ଎Խ • ߴ଎Խ • ϨεϙϯελΠϜΛখ͘͢͞Δ • εϧʔϓοτ (req/s) ͕େ͖͘͢Δ • CPU΍ωοτϫʔΫͳͲϦιʔεফඅྔΛখ͘͢͞Δ • جຊతʹ͸ϨεϙϯελΠϜΛখ͘͢͞Δ͜ͱΛ໨ࢦ͢
10. ISUCONͰ΍Δ͜ͱ • αʔόؚΉ؀ڥߏங • OSɾϛυϧ΢ΣΞͷνϡʔχϯά • ΞϓϦέʔγϣϯϩδοΫͱσʔλߏ଄ͷվળ Ops ※ඞͣ͠΋ҰਓͰશ෦΍Δඞཁ͸ͳ͍
11. ΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ
12. ΦϖϨʔγϣϯͰ఺ΛकΓ ϓϩάϥϛϯάͰ఺ΛͱΔ
13. Ops͸कΓ • OSɾϛυϧ΢ΣΞͷνϡʔχϯά͕ܶతͳՃ఺ཁૉʹ ͳΔ͜ͱ͸͋·Γͳ͍ • OSɾϛυϧ΢ΣΞͷνϡʔχϯά͸ϦιʔεফඅྔΛ খ͘͢͞Δ͜ͱ͕ଟ͍ • େ෯ͳݮ఺ཁૉΛͳ͘͢Πϝʔδ
14. Dev͸߈Ί • σʔλߏ଄ͷվળ͸ࠜຊྍ๏ • Ωϟογϡ/ΦϯϝϞϦ͸ຑༀ
15. Ops͸࢘ྩౝ • Ops͸γεςϜશମΛ၆ᛌ͢Δ • ঢ়گ൑அͱํ਑ܾఆ • ΞΫηεϩά΍αʔόͷෛՙঢ়گΛղੳ • ϘτϧωοΫಛఆ
16. ΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ
17. ISUCONͰ΍Δ͜ͱ • αʔόؚΉ؀ڥߏங • OSɾϛυϧ΢ΣΞͷνϡʔχϯά • ΞϓϦέʔγϣϯϩδοΫͱσʔλߏ଄ͷվળ Ops
18. ؀ڥߏங • Ϋϥ΢υ؀ڥηοτΞοϓ (༧બͷ৔߹) • ISUCON4:AWS ISUCON5:GCE ISUCON6:Azure • ΞΧ΢ϯτ࡞੒ͱΠϯελϯε࡞੒ • αʔόϩάΠϯηοτΞοϓ • ϩάΠϯϢʔβ࡞੒ • ϝϯόʔશһͷsshͷެ։伴Λઃఆ • ͓୊ΞϓϦέʔγϣϯͷىಈ • ϛυϧ΢ΣΞͷΠϯετʔϧ • σϓϩΠͷࣗಈԽ
19. ϘτϧωοΫղੳ • nginxͰΞΫηεϩάΛग़ྗ => URL͝ͱʹղੳ • MySQLͰεϩʔΫΤϦϩά => pt-query-digestͰղੳ • ϓϩϑΝΠϥͰίʔυϨϕϧͰ࣮ߦ࣌ؒղੳ • Perlͷ৔߹ Devel::NYTProf
20. Opsνϡʔχϯά • OS • MySQL • nginx • memcached • redis • ΞϓϦέʔγϣϯͷWebαʔό·ΘΓ
21. ΞϓϦέʔγϣϯ֬ೝ • Opsʹͱͬͯ͸݁ߏݟམͱ͕ͪ͠ • ΞϓϦέʔγϣϯڍಈɺΞϓϦέʔγϣϯϩδοΫɺ σʔλߏ଄ • ؾ͍ͮͨΒɺԿͷΞϓϦέʔγϣϯ͔஌Βͣʹ࡞ۀͯ͠ ͍ͨ • MySQLͷίϯιʔϧʹೖͬͯɺ • ΞϓϦέʔγϣϯ୲౰͕֬ೝ͍ͯ͠Δ͸ͣͳͷͰڭ͑ͯ ΋Β͏͔ɺ࠷ॳʹΈΜͳͰ֬ೝ͢Δ
22. ΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ
23. σϓϩΠͷࣗಈԽ • ඞͣεΫϦϓτΛ༻ҙ͢ΔɻࡶͳͷͰΑ͍ɻ • Slack௨஌ • git pull • carton install (bundle install) • systemctl restart mysql • systemctl restart isuxi.perl • systemctl restart nginx
24. deploy.sh HOSTS=("$@") USERNAME=$USER for HOST in ${HOSTS[@]}; do ssh isucon@$HOST "/home/isucon/notify.sh $USERNAME '$HOST deploying...' && cd /home/isucon/deploy && git pull && ~/deploy/env.sh carton install && sudo systemctl restart mysql && sudo service memcached restart && sudo systemctl restart isuxi.perl && sudo systemctl restart nginx && sudo sysctl -p && /home/isucon/notify.sh $USERNAME '$HOST deploy done'" done
25. ؂ࢹ • ؾ͍ͮͨΒΞϓϦέʔγϣϯ͕མͪͯͯɺؒҧ͑ͯϕϯ νϚʔΫ࣮ߦ • ϕϯνϚʔΫΛ͙͢ʹ࣮ߦͰ͖ͳ͍৔߹͕͋Δ • MackerelͰ֎ܗ؂ࢹͯ͠མͪͯͳ͍͔νΣοΫ • Slack௨஌Ͱ͖ΔͷͰؾ͚ͮͯศར
26. ϛυϧ΢ΣΞͷΠϯετʔϧ • σϑΥϧτͰnginxɺMySQL͕Πϯετʔϧ͞Ε͍ͯΔ ͜ͱ͕ଟ͍ • ࢖͏Մೳੑ͕ߴ͍ͷͰredis΍memcachedΛΠϯετʔ ϧ͢Δ • ιʔε͔ΒϏϧυͯ͠΋Α͍͕ɺόʔδϣϯʹͩ͜ΘΒ ͳ͍ͷͳΒ͹ɺapt-get Ͱ͍ΕΕ͹Α͍
27. Ϧιʔε࢖༻཰೺Ѳ • http://blog.yuuk.io/entry/linux-server-operations • ίΞ͝ͱͷCPUར༻཰ɺσΟεΫI/OɺϝϞϦ࢖༻ྔ etc • ϘτϧωοΫಛఆͷώϯτʹͳΔ͜ͱ͕͋Δ
28. ΞΫηεϩάͷղੳ • proxy (eg.nginx) ͰΞΫηεϩάऔಘ • ϨεϙϯελΠϜ΍ϦΫΤετURL͕ͱΕΔ • alp (https://github.com/tkuchiki/alp) ͳͲͰूܭ • ࣗ࡞εΫϦϓτ https://gist.github.com/yuuki/129983ab4b02e3a646ad • URL͝ͱͷɺϦΫΤετճ਺ɺ߹ܭ࣮ߦ࣌ؒɺฏۉ࣮ߦ࣌ؒͳͲग़ྗ isucon@isucon01:~$ sudo parse_axslog isucon5.access_log.tsv taken_sec req:GET / HTTP/1.1 taken_sum:474.08 req_count:714 avg_taken:0.66 req:GET /footprints HTTP/1.1 taken_sum:58.378 req_count:198 avg_taken:0.29 req:GET /friends HTTP/1.1 taken_sum:27.047 req_count:238 avg_taken:0.11 req:POST /diary/entry HTTP/1.1 taken_sum:6.51 req_count:195 avg_taken:0.03 …
29. MySQLεϩʔΫΤϦϩάͷղੳ • ͋Δᮢ஋Ҏ্ͷ࣮ߦ࣌ؒͷΫΤϦΛग़ྗ • long_query_time = 0 Ͱ͢΂ͯͷΫΤϦΛग़ྗ • pt-query-digestͰूܭ # Profile # Rank Query ID Response time Calls R/Call V/M Item # ==== ================== ============= ===== ====== ===== =============== # 1 0x75B7218DD44BDBC5 27.6810 24.2% 63 0.4394 0.14 SELECT relations # 2 0x492C7CAB538DE80D 22.5882 19.7% 53 0.4262 0.25 SELECT footprints # 3 0xA44D49C56E635213 14.5630 12.7% 63 0.2312 0.02 SELECT entries # 4 0x57EBD74C3B600DB8 14.0403 12.3% 40 0.3510 0.02 SELECT entries # 5 0xDC67C8CE8419EE5B 12.8485 11.2% 7 1.8355 0.19 SELECT entries
30. OSͷνϡʔχϯά $ cat /etc/sysctl.d/99-sysctl.conf net.ipv4.tcp_max_tw_buckets = 2000000 net.ipv4.ip_local_port_range = 10000 65000 net.core.somaxconn = 32768 net.core.netdev_max_backlog = 8192 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 10 fs.file-max = 100000 Systemdͷ৔߹ɺόʔδϣϯ207ͱ21x͔Β/etc/sysctl.conf͕ ແޮ
31. nginxͷνϡʔχϯά worker_processes 1; worker_rlimit_nofile 65535; events { worker_connections 8192; } http { include mime.types; sendfile on; keepalive_timeout 120s; tcp_nopush on; tcp_nodelay on; etag off;
32. MySQLͷνϡʔχϯά innodb_buffer_pool_size = 1G innodb_flush_log_at_trx_commit = 0 innodb_flush_method=O_DIRECT
33. PerlΞϓϦέʔγϣϯαʔόνϡʔχϯά start_server --backlog 16384 -- plackup -s Gazelle --max-reqs-per-child 50000 -workers=4 • ϫʔΧʔϓϩηε਺͸ίΞ਺ͱಉఔ౓ʹ͢Δ • ͜ΕҎ্૿΍ͯ͠΋ίϯςΩετεΠον૿͑Δ͚ͩ ͰείΞ͸৳ͼͳ͍͜ͱ͕ଟ͍ • preforkܕαʔόͰͳΔ΂͘fork͠ͳ͓ͤ͞ͳ͍Α͏ʹ
34. memcachedͷνϡʔχϯά -m 256 -U 0 -c 20000 -v -t 1 -C -B ascii
35. Redisͷνϡʔχϯά tcp-keepalive 60 save 900 1 save 300 10 save 120 20000 maxclients 10000 appendonly yes appendfsync everysec no-appendfsync-on-rewrite no
36. UnixυϝΠϯιέοτ • ༧બͰ͸1୆ͷαʔό্ʹɺ֤ϛυϧ΢ΣΞ͕ಉډ • proxy <=> app <=> db ͷ઀ଓ͕TCPͩͱ΋͍ͬͨͳ͍ • 3-way handshakeͦͷଞͷΦʔόϔου • ΦʔόϔουΛͳͨ͘͢ΊUnixυϝΠϯιέοτΛ࢖͏ start_server --path /dev/shm/app.sock upstream app { server unix:/dev/shm/app.sock; } nginx <=> Plack ͷ৔߹
37. http://www.slideshare.net/kazeburo/isucon-summerclass2014action2final
38. ΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ
39. ·ͱΊ • ISUCONͷΦϖϨʔγϣϯʹ͓͚Δయܕతͳܕͷ1ͭΛ ঺հ • ISUCONͰ͸ϨεϙϯελΠϜΛͰ͖Δ͚ͩখ͘͢͞Δ • Ops͸कΓɺDev͸߈Ίͷݪଇ
40. Opsͷ͍͍ͱ͜Ζ • ൚༻తͳ஌ࣝ΍ΞϓϩʔνͰઓ͑Δ • શମΛΈͯΔײΛຯΘ͑Δ • ࣄલͷ࿅श͕΍Γ΍͍͢
41. ISUCON6༧બ 9݄17೔(౔) ,18೔(೔)