༻్ͱίεύͰߟ͑Δ ϋʔυ΢ΣΞબͼͷίπ 2017/03/17 pixiv night in Fukuoka Πϯϑϥ෦ ೆ઒ ؽจ

ࣗݾ঺հ ▸ ໊લ: ೆ઒ ؽจ ▸ pixiv 6೥໨ ▸ Πϯϑϥ෦Ͱpixivͱؔ࿈αʔϏεશମͷӡ༻Λ΍͍ͬͯ·͢ ▸ αʔόͷௐୡ͔ΒηοτΞοϓɺϛυϧ΢ΣΞͷӡ༻ɺ؀ڥߏஙɺΞϓϦέʔ γϣϯͷνϡʔχϯάɺΫϥ΢υपΓͳͲ͍Ζ͍Ζ ▸ pixivΠϯϑϥ෦ 4ਓ͘Β͍Ͱճ͍ͯ͠Δ͏ͪͷҰਓͰ͢ ▸ ೔ࠒ͸ιϑτ΢ΣΞΛ৮Δ͜ͱ͕ଟ͍ͷͰ͕͢ɺࠓ೔͸ϋʔυͷ࿩Λ͖࣋ͬͯ ·ͨ͠ ▸ pixivͰ͸elasticsearch, RiakCS, RedisͳͲΛಛʹ৮͍ͬͯΔ͜ͱ͕ଟ͍Ͱ͢ ▸ Ruby, PHP, JavaͰίʔυΛॻ͘͜ͱ͕͋Γ·͢

QJYJWͷαʔόͨͪ ▸ ৽॓DC 200୆ະຬ ▸ pixiv.netΛத৺ͱͨ͠αʔό͕Քಇ ▸ ΞϓϦέʔγϣϯɺDBɺը૾ετϨʔδͳͲ ▸ നՏDC 100୆ະຬ ▸ ը૾ΩϟογϡαʔόΛத৺ ▸ comic.pixiv.net ͷΞϓϦέʔγϣϯɺσʔλϕʔεͳͲ ▸ ৽॓DCͱґଘ͕௿͍΋ͷΛߏங ▸ ։ൃػɺΫϥ΢υ ਺े୆ ▸ ݕূ༻ͳͲ୹ظతʹ࢖͏αʔό

Ϋϥ΢υ ▸ ௕ظӡ༻͠ͳ͍ɺݕূ΍ظؒݶఆΠϕϯτ༻ΞϓϦέʔγϣϯ ▸ ݕূ؀ڥ͸ΦϯϓϨΛ༻ҙ͠ͳͯ͘΋े෼ͳ͜ͱ͕͋Δ ▸ ͍҆ΦϯϓϨ؀ڥ͸ผͷ༻్ʹ࢖ͬͯػձଛࣦΛ཈͑Δ ▸ ΦϯσϚϯυͰ༻ҙͰ͖Δར఺ > ΦϯϓϨ͸͕͔͔࣌ؒΔ ▸ όοΫΞοϓ ▸ ߋ৽ස౓͕௿͍σʔλͷόοΫΞοϓ ▸ ผڌ఺ʹΑΔ৑௕Խ ▸ DCΛผڌ఺ʹ༻ҙ͢ΔΑΓ͸͍҆ (DNSͳͲ)

Ϋϥ΢υ ▸ సૹྔ͕গͳ͍ ▸ ࣌ؒ vs අ༻ Ͱ࣌ؒ୹ॖͷޮՌ͕େ͖͍΋ͷ ▸ ϩάղੳ ▸ ظ͕ؒ௕͘ܭࢉ͕࣌ؒ௕͘ͳΔ ͔ͭ ॏཁͳࢦඪ ͸֎෦αʔϏ εʹ౤͛ͯ΋͍҆͜ͱ͕͋Δ ▸ ୹ظؒɺಛʹैྔ՝ۚͷ΄͏͕҆͘ͳΔέʔεͰ͸Ϋϥ΢υΛར ༻ ▸ ΦϯϓϨ͸௕ظతʹ͸҆͘ͳΔͷͰɺ਺ϲ݄Ҏ্࢖͏΋ͷ΍ɺ௕ ظӡ༻ʹҠߦ͢Δ৔߹͸Ϋϥ΢υ͔ΒҾ্͖͛

ΦϯϓϨαʔό ▸ αʔόߏ੒Λߟ͑ͯઃஔ ▸ CPUɺϝϞϦɺσΟεΫIOͳͲ༻్ʹ߹ΘͤͯύʔπΛ࠷ దԽ͍ͯ͘͠ ▸ ΞϓϦέʔγϣϯ/σʔλϕʔε/ετϨʔδ Λྫʹύʔπબ ͼͷྫΛڍ͍͖͛ͯ·͢ɻ

αʔό͝ͱͷ܏޲

ΞϓϦέʔγϣϯαʔό ▸ HTTPϦΫΤετΛॲཧ͢ΔWebαʔό ▸ Apache ΍ Nginx ▸ PHP ▸ Ruby ▸ Java ͳͲ ▸ ΞϓϦέʔγϣϯͷઃܭʹ߹Θͤͯ࠷దԽ͢Δඞཁ͋Γ

ΞϓϦέʔγϣϯαʔό ▸ pixiv.net ͸ PHP(Apache + mod_php) Ͱಈ࡞ ▸ prefork (Ϛϧνϓϩηε) ▸ Apacheͷ֤ϓϩηεΛͦΕͧΕCPUίΞͰಈ͔͢ (γϯ άϧεϨουɾϚϧνϓϩηε) ▸ CPUίΞ͕ଟ͍΄͏͕ฒྻੑ͸ߴ͍

ΞϓϦέʔγϣϯαʔό ▸ BOOTH/pixivFACTORY ͳͲ Ruby (RoR) ͷαʔό ▸ nginx + unicorn Ͱಈ࡞ ▸ ApacheͷPreforkϞσϧͱ͍ۙ ▸ unicorn͸MasterʹΞϓϦέʔγϣϯίʔυΛಡΈࠐΜͩ͋ͱ fork͢Δ ▸ ࠩ෼͚ͩϝϞϦΛফඅ͢Δ CoW ͕ͩͦΕͳΓʹϝϞϦΛ࢖͏ ▸ PHPαʔόͱൺֱͯ͠ϝϞϦ࢖༻཰͕ߴ͍ͨΊϝϞϦ΋CPUͷ ڧ͍αʔόʹ͢Δඞཁ͕͋Δ

ΞϓϦέʔγϣϯαʔό ▸ Javaαʔό ▸ Solr΍ElasticsearchͳͲͷݕࡧΤϯδϯͰ͸ͳ͘JettyͳͲͰಈ͘Webαʔό ▸ ಡΈࠐΉσʔλ͸গͳ͍͕ɺΞϓϦέʔγϣϯΩϟογϡΛ࣋ͭඞཁ͕͋ Δέʔε͕͋Δ ▸ ϝϞϦ͸PHPαʔόͱಉ͡ఔ౓ͰΑ͍ ▸ εϨουΛ࢖ͬͨฒྻॲཧͰ͸CPUੑೳΛ׆͔͠΍͘͢CPUίΞ਺ͷଟ͍Ϛ γϯͰߴ଎ԽͰ͖Δ ▸ ϝϞϦׂΓ౰ͯΛେ͖͘͠ա͗Δͱ GC Ͱࢭ·Γ͕ͪʹͳΔ ▸ GCઓུ͸CMSͱ͍ͨ͠ͷͰCPUੑೳ͸͋ͬͨ΄͏͕Α͍

σʔλϕʔεαʔό ▸ MySQL ▸ InnoDB όοϑΝϓʔϧʹଟ͘ͷσʔλΛࡌͤΔ ▸ σΟεΫ΁ͷΞΫηε͕গͳ͘ͳΔΑ͏ʹ͢Δ ▸ bin/relaylogͷॻ͖ࠐΈ͕஗͘ͳΒͳ͍Α͏ߴ଎ͳSSDΛ datadirʹࢦఆ ▸ ΫΤϦ͕࠷దԽ͞Ε͍ͯΕ͹CPUੑೳ͸ͦΕ΄Ͳཁٻ͞ Εͳ͍

σʔλϕʔεαʔό ▸ Solr / Elasticsearch ▸ Javaͷӡ༻͸APͱࣅ͍ͯΔ ▸ σʔλͷϩʔυɺωοτϫʔΫ௨৴ɺώʔϓαΠζ͸େ͖͍ ▸ ϝϞϦׂΓ౰ͯΛେ͖͘͠ա͗Δͱ GC Ͱࢭ·Γ͕ͪʹͳΔ ▸ GCઓུ͸CMSͱ͍ͨ͠ͷͰCPUੑೳ͸͋ͬͨ΄͏͕Α͍ ▸ 32GBҎ্ͷϝϞϦΛJVMͷώʔϓʹׂΓ͋ͯΔ৔߹ JVM ͕࢖͏ϙΠϯλ αΠζ͕4όΠτ͔Β8όΠτʹͳΓޮ཰͕ѱ͘ͳΔ ▸ 1ϊʔυ͋ͨΓ32GBҎ্ͷώʔϓׂΓ౰ͯʹͳΒͳ͍Α͏ʹ͢Δ ▸ ΠϯσοΫεͷ୳ࡧʹ͸CPUΛ࢖͏ͷͰCPU΋ڧ͍΋ͷΛ࢖͏

σʔλϕʔεαʔό ▸ Redis ▸ ΦϯϝϞϦσʔλϕʔε ▸ CPUͷෳ਺ίΞΛར༻Ͱ͖ͳ͍ ▸ ίΞ਺͕ଟ͍΋ͷ͸ඞཁ͕ͳ͍ ▸ σʔλͷΞΫηεྔʹ߹ΘͤͯCPUΛ༻ҙ ▸ ϝϞϦ͸σʔλྔʹ߹Θͤͯ༻ҙ ▸ σʔλͷμϯϓɺϩʔυΛߴ଎ʹ͢ΔͨΊSSDΛ༻ҙ

ετϨʔδαʔό ▸ ը૾αʔό ▸ αϜωΠϧͱΦϦδφϧαΠζͷը૾ɺछྨʹΑͬͯΞΫηε܏ ޲͕ҟͳΔ ▸ ΞΫηε͕ଟ͍খ͞ͳϑΝΠϧΛૉૣ͘ฦ͢ʹ͸SSDͷΑ͏ͳ ϥϯμϜΞΫηεੑೳͷߴ͍σΟεΫΛ࢖͏ඞཁ͕͋Δ ▸ ΦϦδφϧαΠζͷը૾ɺաڈͷΠϥετ͸ΞΫηεස౓͕௿ ͘ͳΔ܏޲͕͋ΓɺHDDͰ΋े෼ͳ͜ͱ͕͋Δ ▸ αΠζ΍ΞΫηε܏޲ʹΑͬͯɺSSDͱHDDΛ࢖͍෼͚Δ͜ͱ ͰίετύϑΥʔϚϯεΛΑ͍ͯ͘͠Δ

ετϨʔδαʔό ▸ Riak / RiakCS ▸ ෼ࢄετϨʔδαʔό ▸ BOOTH/pixivFACTORYͳͲͷαʔϏεͰར༻ ▸ ৑௕Խ͞Ε͍ͯΔͷͰɺHDDΛRAID0ͰଋͶͯετϨʔδͱ͢Δ ▸ vnode୯ҐͰϝϞϦʹࡌͬͨΓɺbitcaskͷΩʔҰཡ͕ϝϞϦʹऩ ·ΔΑ͏ʹ͠ͳ͚Ε͹ͳΒͳ͍ ▸ ϝϞϦͷཁٻ͸σʔλαΠζʹΑͬͯ૿͑Δ ▸ ߴ଎ͳϨεϙϯεʹ͸΍΍޲͔ͳ͍͕ɺେ͖ͳϑΝΠϧΛѻ͍΍ ͍͢Α͏ʹͳ͍ͬͯΔ

ϋʔυ΢ΣΞͷબͼํ

ϋʔυ΢ΣΞ$16 ▸ Intel CPU Xeon E5 2600v4 Λ࢖͏͜ͱ͕ଟ͍ ▸ ௿ίετ > E5 2603v4 2609v4 ▸ CPUΛ࢖Θͳ͍έʔε ▸ தύϑΥʔϚϯε > E5 2620v4 ʙ 2640v4 ▸ CPUΛ࢖͏έʔε ▸ ߴप೾਺CPU(E5 2637v4ͳͲ) ΍ ͦΕҎ্ͷੑೳ ▸ ར༻͸ࠓͷͱ͜Ζͳ͠

ϋʔυ΢ΣΞϝϞϦ ▸ DDR3 ͔Β DDR4 ΁ͷஔ͖׵్্͑ ▸ ൒೥લ͘Β͍͸ϝϞϦՁ͕֨ඇৗʹ͔҆ͬͨ ▸ ݱࡏɺԁ҆ɾڙڅෆ଍ʹΑΓߴಅத ▸ ϝϞϦՁ֨͸ϒϨ͕େ͖͘Ձ֨ͷݟఆΊ͸೉͍͠ ▸ DBαʔόͰͷϝϞϦར༻͕ଟ͍ ▸ ௚ۙͰ͸32GBϞδϡʔϧத৺Ͱར༻

ϋʔυ΢ΣΞετϨʔδ)%% ▸ ίϯγϡʔϚʔ޲͚SATA HDDΛଋͶΔͱίετ͸͍҆ ▸ ਺͕ग़͍ͯΔͨΊ ▸ ݕূσʔλ΋ଟ͍ͨΊࢀߟʹͳΓ΍͍͢ ▸ ΤϯλʔϓϥΠζ΍ϋΠΤϯυϞσϧ͸ߴՁ͕ͩੑೳ໘Ͱ༏ΕΔ͜ͱ΋ ▸ ݕূ༻ʹগྔࢼ͢͜ͱ͸͋Δ͕ຊӡ༻Ͱͷར༻͸ࠓͷͱ͜Ζͳ͍ ▸ HGST / WesternDigital / TOSHIBA / Seagate ͳͲ͍Ζ͍Ζͳϝʔ Χʔͷ΋ͷΛར༻ ▸ ϩοτʹΑΓނো཰͕ҧ͏܏޲͕Θ͔͍ͬͯΔͨΊɺϩοτ͝ͱʹӡ ༻Λ෼͚͍ͯΔ

ϋʔυ΢ΣΞετϨʔδ44% ▸ ίϯγϡʔϚʔ޲͚SATA SSDΛ࢖͏έʔε͕΄ͱΜͲ ▸ HDDͱಉ͘͡਺͕ग़͍ͯΔͨΊ ▸ Ұ෦αʔό༻్޲͚SSDΛ࢖͍ͬͯΔ৔ॴ΋͋Δ ▸ Samsung / Crucial / Intel ͳͲ͍Ζ͍ΖͳϝʔΧʔͷ΋ͷΛ ར༻

ϋʔυ΢ΣΞ)%%WT44% ▸ SSDΛ࢖͏͔Ͳ͏͔͸IOੑೳͱهԱ༰ྔɺՁ֨໘Ͱ༏ΕΔ͔Ͳ͏͔ ▸ هԱ༰ྔ/Ձ֨Ͱ͸ HDD > SSD SATA > SSD m.2 ▸ ௚ۙSSDՁ֨͸͔ͳΓ҆͘ͳ͍ͬͯΔ ▸ HDD͸େ༰ྔԽ͸ਐΜͰ͍Δ͕ɺIOੑೳ໘Ͱ೉͕͋Δ͜ͱ΋͋Δɻ ▸ γʔέϯγϟϧΞΫηε༻్Ͱ͸௿ίετͳͨΊ࠾༻͍ͯ͠Δ ▸ SSDͰ΋ಉ͡Ͱɺେ༰ྔԽʹ߹ΘͤͯIOੑೳ΋্͕͍ͬͯ͘ͷ͕ཧ૝ ▸ SATAͷੑೳݶքΛ௒͑ΔͨΊ௚ۙͰ͸m.2ΠϯλʔϑΣΠεϞσϧ Λར༻ (༧ఆ)

·ͱΊ ▸ pixivͰ͸ΦϯϓϨ؀ڥΛ͏·͘׆༻ͯ͠ίετύϑΥʔϚ ϯεͷߴ͍αʔόӡ༻Λ͍ͯ͠·͢ ▸ ௿Ձ֨Ͱ΋े෼ੑೳΛಘΔͨΊɺ༻్ʹ߹Θ༷ͤͯʑͳ޻ ෉Λ͍ͯ͠·͢ ▸ ϐΫγϒͰ͸ΠϯϑϥΤϯδχΞΛืू͍ͯ͠·͢