ISUCON༧બಥഁΛࢧ͑Δ ΦϖϨʔγϣϯٕज़ id:y_uuki Kyoto.ͳΜ͔ #2

ISUCON ͓୊ͱͳΔWebαʔϏεΛܾΊΒΕͨϨΪϡϨʔγϣ ϯͷதͰݶք·Ͱߴ଎ԽΛਤΔνϡʔχϯάότϧ

ISUCONॳग़৔ͷͱ͖ʹ ࣗ෼͕஌Γ͔ͨͬͨ͜ͱ

w/id:motemen id:Songmu ຊઓग़৔ => ࢂഊ

w/id:astj id:ntakanashi ༧બ શମ5Ґ ຊઓग़৔ => ࢂഊ

No content

΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ

΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ

ISUCONͰ΍Δ͜ͱ • ༩͑ΒΕͨ΢ΣϒΞϓϦέʔγϣϯΛͱʹ͔͘ߴ଎Խ • ߴ଎Խ • ϨεϙϯελΠϜΛখ͘͢͞Δ • εϧʔϓοτ (req/s) ͕େ͖͘͢Δ • CPU΍ωοτϫʔΫͳͲϦιʔεফඅྔΛখ͘͢͞Δ • جຊతʹ͸ϨεϙϯελΠϜΛখ͘͢͞Δ͜ͱΛ໨ࢦ͢

ISUCONͰ΍Δ͜ͱ • αʔόؚΉ؀ڥߏங • OSɾϛυϧ΢ΣΞͷνϡʔχϯά • ΞϓϦέʔγϣϯϩδοΫͱσʔλߏ଄ͷվળ Ops ※ඞͣ͠΋ҰਓͰશ෦΍Δඞཁ͸ͳ͍

΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ

ΦϖϨʔγϣϯͰ఺ΛकΓ ϓϩάϥϛϯάͰ఺ΛͱΔ

Ops͸कΓ • OSɾϛυϧ΢ΣΞͷνϡʔχϯά͕ܶతͳՃ఺ཁૉʹ ͳΔ͜ͱ͸͋·Γͳ͍ • OSɾϛυϧ΢ΣΞͷνϡʔχϯά͸ϦιʔεফඅྔΛ খ͘͢͞Δ͜ͱ͕ଟ͍ • େ෯ͳݮ఺ཁૉΛͳ͘͢Πϝʔδ

Dev͸߈Ί • σʔλߏ଄ͷվળ͸ࠜຊྍ๏ • Ωϟογϡ/ΦϯϝϞϦ͸ຑༀ

Ops͸࢘ྩౝ • Ops͸γεςϜશମΛ၆ᛌ͢Δ • ঢ়گ൑அͱํ਑ܾఆ • ΞΫηεϩά΍αʔόͷෛՙঢ়گΛղੳ • ϘτϧωοΫಛఆ

΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ

ISUCONͰ΍Δ͜ͱ • αʔόؚΉ؀ڥߏங • OSɾϛυϧ΢ΣΞͷνϡʔχϯά • ΞϓϦέʔγϣϯϩδοΫͱσʔλߏ଄ͷվળ Ops

؀ڥߏங • Ϋϥ΢υ؀ڥηοτΞοϓ (༧બͷ৔߹) • ISUCON4:AWS ISUCON5:GCE ISUCON6:Azure • ΞΧ΢ϯτ࡞੒ͱΠϯελϯε࡞੒ • αʔόϩάΠϯηοτΞοϓ • ϩάΠϯϢʔβ࡞੒ • ϝϯόʔશһͷsshͷެ։伴Λઃఆ • ͓୊ΞϓϦέʔγϣϯͷىಈ • ϛυϧ΢ΣΞͷΠϯετʔϧ • σϓϩΠͷࣗಈԽ

ϘτϧωοΫղੳ • nginxͰΞΫηεϩάΛग़ྗ => URL͝ͱʹղੳ • MySQLͰεϩʔΫΤϦϩά => pt-query-digestͰղੳ • ϓϩϑΝΠϥͰίʔυϨϕϧͰ࣮ߦ࣌ؒղੳ • Perlͷ৔߹ Devel::NYTProf

Opsνϡʔχϯά • OS • MySQL • nginx • memcached • redis • ΞϓϦέʔγϣϯͷWebαʔό·ΘΓ

ΞϓϦέʔγϣϯ֬ೝ • Opsʹͱͬͯ͸݁ߏݟམͱ͕ͪ͠ • ΞϓϦέʔγϣϯڍಈɺΞϓϦέʔγϣϯϩδοΫɺ σʔλߏ଄ • ؾ͍ͮͨΒɺԿͷΞϓϦέʔγϣϯ͔஌Βͣʹ࡞ۀͯ͠ ͍ͨ • MySQLͷίϯιʔϧʹೖͬͯɺ • ΞϓϦέʔγϣϯ୲౰͕֬ೝ͍ͯ͠Δ͸ͣͳͷͰڭ͑ͯ ΋Β͏͔ɺ࠷ॳʹΈΜͳͰ֬ೝ͢Δ

΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ

σϓϩΠͷࣗಈԽ • ඞͣεΫϦϓτΛ༻ҙ͢ΔɻࡶͳͷͰΑ͍ɻ • Slack௨஌ • git pull • carton install (bundle install) • systemctl restart mysql • systemctl restart isuxi.perl • systemctl restart nginx

deploy.sh HOSTS=("$@") USERNAME=$USER for HOST in ${HOSTS[@]}; do ssh isucon@$HOST "/home/isucon/notify.sh $USERNAME '$HOST deploying...' && cd /home/isucon/deploy && git pull && ~/deploy/env.sh carton install && sudo systemctl restart mysql && sudo service memcached restart && sudo systemctl restart isuxi.perl && sudo systemctl restart nginx && sudo sysctl -p && /home/isucon/notify.sh $USERNAME '$HOST deploy done'" done

؂ࢹ • ؾ͍ͮͨΒΞϓϦέʔγϣϯ͕མͪͯͯɺؒҧ͑ͯϕϯ νϚʔΫ࣮ߦ • ϕϯνϚʔΫΛ͙͢ʹ࣮ߦͰ͖ͳ͍৔߹͕͋Δ • MackerelͰ֎ܗ؂ࢹͯ͠མͪͯͳ͍͔νΣοΫ • Slack௨஌Ͱ͖ΔͷͰؾ͚ͮͯศར

ϛυϧ΢ΣΞͷΠϯετʔϧ • σϑΥϧτͰnginxɺMySQL͕Πϯετʔϧ͞Ε͍ͯΔ ͜ͱ͕ଟ͍ • ࢖͏Մೳੑ͕ߴ͍ͷͰredis΍memcachedΛΠϯετʔ ϧ͢Δ • ιʔε͔ΒϏϧυͯ͠΋Α͍͕ɺόʔδϣϯʹͩ͜ΘΒ ͳ͍ͷͳΒ͹ɺapt-get Ͱ͍ΕΕ͹Α͍

Ϧιʔε࢖༻཰೺Ѳ • http://blog.yuuk.io/entry/linux-server-operations • ίΞ͝ͱͷCPUར༻཰ɺσΟεΫI/OɺϝϞϦ࢖༻ྔ etc • ϘτϧωοΫಛఆͷώϯτʹͳΔ͜ͱ͕͋Δ

ΞΫηεϩάͷղੳ • proxy (eg.nginx) ͰΞΫηεϩάऔಘ • ϨεϙϯελΠϜ΍ϦΫΤετURL͕ͱΕΔ • alp (https://github.com/tkuchiki/alp) ͳͲͰूܭ • ࣗ࡞εΫϦϓτ https://gist.github.com/yuuki/129983ab4b02e3a646ad • URL͝ͱͷɺϦΫΤετճ਺ɺ߹ܭ࣮ߦ࣌ؒɺฏۉ࣮ߦ࣌ؒͳͲग़ྗ isucon@isucon01:~$ sudo parse_axslog isucon5.access_log.tsv taken_sec req:GET / HTTP/1.1 taken_sum:474.08 req_count:714 avg_taken:0.66 req:GET /footprints HTTP/1.1 taken_sum:58.378 req_count:198 avg_taken:0.29 req:GET /friends HTTP/1.1 taken_sum:27.047 req_count:238 avg_taken:0.11 req:POST /diary/entry HTTP/1.1 taken_sum:6.51 req_count:195 avg_taken:0.03 …

MySQLεϩʔΫΤϦϩάͷղੳ • ͋Δᮢ஋Ҏ্ͷ࣮ߦ࣌ؒͷΫΤϦΛग़ྗ • long_query_time = 0 Ͱ͢΂ͯͷΫΤϦΛग़ྗ • pt-query-digestͰूܭ # Profile # Rank Query ID Response time Calls R/Call V/M Item # ==== ================== ============= ===== ====== ===== =============== # 1 0x75B7218DD44BDBC5 27.6810 24.2% 63 0.4394 0.14 SELECT relations # 2 0x492C7CAB538DE80D 22.5882 19.7% 53 0.4262 0.25 SELECT footprints # 3 0xA44D49C56E635213 14.5630 12.7% 63 0.2312 0.02 SELECT entries # 4 0x57EBD74C3B600DB8 14.0403 12.3% 40 0.3510 0.02 SELECT entries # 5 0xDC67C8CE8419EE5B 12.8485 11.2% 7 1.8355 0.19 SELECT entries

OSͷνϡʔχϯά $ cat /etc/sysctl.d/99-sysctl.conf net.ipv4.tcp_max_tw_buckets = 2000000 net.ipv4.ip_local_port_range = 10000 65000 net.core.somaxconn = 32768 net.core.netdev_max_backlog = 8192 net.ipv4.tcp_tw_reuse = 1 net.ipv4.tcp_fin_timeout = 10 fs.file-max = 100000 Systemdͷ৔߹ɺόʔδϣϯ207ͱ21x͔Β/etc/sysctl.conf͕ ແޮ

nginxͷνϡʔχϯά worker_processes 1; worker_rlimit_nofile 65535; events { worker_connections 8192; } http { include mime.types; sendfile on; keepalive_timeout 120s; tcp_nopush on; tcp_nodelay on; etag off;

MySQLͷνϡʔχϯά innodb_buffer_pool_size = 1G innodb_flush_log_at_trx_commit = 0 innodb_flush_method=O_DIRECT

PerlΞϓϦέʔγϣϯαʔόνϡʔχϯά start_server --backlog 16384 -- plackup -s Gazelle --max-reqs-per-child 50000 -workers=4 • ϫʔΧʔϓϩηε਺͸ίΞ਺ͱಉఔ౓ʹ͢Δ • ͜ΕҎ্૿΍ͯ͠΋ίϯςΩετεΠον૿͑Δ͚ͩ ͰείΞ͸৳ͼͳ͍͜ͱ͕ଟ͍ • preforkܕαʔόͰͳΔ΂͘fork͠ͳ͓ͤ͞ͳ͍Α͏ʹ

memcachedͷνϡʔχϯά -m 256 -U 0 -c 20000 -v -t 1 -C -B ascii

Redisͷνϡʔχϯά tcp-keepalive 60 save 900 1 save 300 10 save 120 20000 maxclients 10000 appendonly yes appendfsync everysec no-appendfsync-on-rewrite no

UnixυϝΠϯιέοτ • ༧બͰ͸1୆ͷαʔό্ʹɺ֤ϛυϧ΢ΣΞ͕ಉډ • proxy <=> app <=> db ͷ઀ଓ͕TCPͩͱ΋͍ͬͨͳ͍ • 3-way handshakeͦͷଞͷΦʔόϔου • ΦʔόϔουΛͳͨ͘͢ΊUnixυϝΠϯιέοτΛ࢖͏ start_server --path /dev/shm/app.sock upstream app { server unix:/dev/shm/app.sock; } nginx <=> Plack ͷ৔߹

http://www.slideshare.net/kazeburo/isucon-summerclass2014action2final

΋͘͡ 1. ISUCONͰ΍Δ͜ͱ 2. ISUCONͷߟ͑ํ 3. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (جຊฤ) 4. ISUCONʹ͓͚ΔΦϖϨʔγϣϯ (࣮ફฤ) 5. ·ͱΊ

·ͱΊ • ISUCONͷΦϖϨʔγϣϯʹ͓͚Δయܕతͳܕͷ1ͭΛ ঺հ • ISUCONͰ͸ϨεϙϯελΠϜΛͰ͖Δ͚ͩখ͘͢͞Δ • Ops͸कΓɺDev͸߈Ίͷݪଇ

Opsͷ͍͍ͱ͜Ζ • ൚༻తͳ஌ࣝ΍ΞϓϩʔνͰઓ͑Δ • શମΛΈͯΔײΛຯΘ͑Δ • ࣄલͷ࿅श͕΍Γ΍͍͢

ISUCON6༧બ 9݄17೔(౔) ,18೔(೔)