ピクシブ社内広告サーバーでのGoの開発・運用 #gocon /p_ads_server_gocon2015

by Kaneko Tatsuya

Slide 1

Slide 1 text

ϐΫγϒࣾ಺޿ࠂαʔόʔͰͷ Goͷ։ൃɾӡ༻ ϐΫγϒ Πϯϑϥ catatsuy

Slide 2

Slide 2 text

About me • catatsuyͱ͍͏IDͰΠϯλʔωοτ্Ͱ׆ಈ • ΧλπΠͱ͔ͬͯݺ͹Ε͍ͯΔ • ϐΫγϒגࣜձࣾͰΠϯϑϥΛ୲౰ • ޿ࠂαʔόʔ͕ଟ͍͕ଞʹ΋৭ʑ • एखΠϯϑϥͱͯ͠ͷ࿩ʹ͍ͭͯJTF2015Ͱ࠲ஊձܗࣜͰ࿩͢༧ఆ • ৽ॻͰʰpixivΤϯδχΞ͕ڭ͑Δϓϩάϥϛϯάೖ໳ʱͱ͍͏ຊΛग़͠·ͨ͠ • ϐΫγϒࣾ಺ͷඇΤϯδχΞ޲͚ʹ࡞ͬͨݚम಺༰ͷॻ੶Խ • ࠓճ͸ࣾ಺ͷ޿ࠂαʔόʔͰͷGoͷࣄྫΛ঺հ

Slide 3

Slide 3 text

DNS ޿ࠂ഑৴αʔόʔ Admin MySQL qVFOUE NGINX IMAGE CACHE LTLMPH NFNDBDIFE Ad data "QQMJDBUJPO (PMBOH Impressions Clicks $JSDVT Ad data Aggregate results Round Robin 6/*9%PNBJO4PDLFU 'JMFEFTDSJQUPS 6/*9%PNBJO4PDLFU type tail

Slide 4

Slide 4 text

Nginx and Go • ϢʔβʔͷϦΫΤετΛ௚઀Go͕ड͚ΔͷͰ͸ͳ͘NginxͰड͚͍ͨ • ϢʔβʔͷϦΫΤετΛόοϑΝϦϯάͯ͘͠ΕΔͷͰίωΫγϣϯͷอ࣋ͳͲ ໘౗ͳ͜ͱΛߟ͑Δඞཁ͕ͳ͘ͳΔ • GoͰ͸௿஗ԆͰϨεϙϯεΛฦ͢͜ͱ͚ͩΛߟ͑Ε͹Α͍ • ΞΫηε੍ޚͳͲ΋؆୯ • ΩϟογϡͷػߏͳͲ͕؆୯ʹ࢖͑Δ • GoͰtcpΛlistenͯ͠ىಈͨ͠αʔόʔʹରͯ͠proxy_pass͢Δͱ஗͔ͬͨ • ύέοτͷαΠζ͕খ͍͞ͷͰtcpͷ3way handshakeͷίετ͕ߴ͔ͬͨ • UNIX domain socketΛlistenͯ͠αʔόʔΛىಈ͍ͨ͠

Slide 5

Slide 5 text

Circus • Mozilla͕࡞ͬͨPython੡ͷϓϩηε؅ཧπʔϧ • Circusࣗମ͕ιέοτϑΝΠϧΛ࣋ͬͯɼΞϓϦέʔγϣϯ͸ϑΝΠϧ σΟεΫϦϓλΛListenͤͯ͞ىಈ͢Δ • σϓϩΠ࣌ʹ৽͍͠ϓϩηε͕ىಈ͔ͯ͠Βݹ͍ϓϩηεΛࡴ͢ • μ΢ϯλΠϜͳ͠ͰΞϓϦέʔγϣϯΛೖΕସ͑Δ • circusctlͱ͍͏ίϚϯυπʔϧܦ༝Ͱૢ࡞͢Δ • σϓϩΠ͸όΠφϦΛrsyncͯ͠circusctl reload͢Δ͚ͩʹ • stdout/stderrͷϩάϑΝΠϧԽͱϩάϩʔςʔτ΋ઃఆ࣍ୈͰՄೳ

Slide 6

Slide 6 text

UNIX domain socket • ಉҰϚγϯ্ͳΒ਌ࢠؔ܎͕ແͯ͘΋Ͱ͖Δϓϩηεؒ௨৴ • memcachedͱͷ௨৴ʹ΋࢖༻͢Δͱߴ଎͔ͭ҆ఆతʹಈ࡞ͤ͞Δ ͜ͱ͕Ͱ͖Δ • CircusΛ࢖͏ͳΒιέοτϑΝΠϧ͸Circus͕࡞੒͢ΔͷͰGoଆͰ͸ ϑΝΠϧσΟεΫϦϓλΛListen͢ΔαʔόʔΛ࣮૷͢Δඞཁ͕͋Δ • ։ൃΛ༰қʹ͢ΔͨΊʹtcpͰ΋ListenͰ͖ΔΑ͏ʹ͍ͨ͠ • net.ListenerΛ༻ҙ͢Ε͹෼ذΛগͳ͘Ͱ͖Δ • net.Listen("unix", path) / net.ListenTCP / net.FileListener

Slide 7

Slide 7 text

Profiling • ຊ൪౤ೖલʹͲ͜Ͱ͕͔͔࣌ؒΔͷ͔ݟ͓͖͍ͯͨ • github:davecheney/profile • ͕͔͔࣌ؒΔؔ਺ͳͲΛՄࢹԽ͢ΔPDFϑΝΠϧ΋ग़ྗͰ͖Δ • Introducing profile, super simple profiling for Go programs | Dave Cheney • h[p:/ /dave.cheney.net/2013/07/07/introducing-‐profile-‐super-‐simple-‐ profiling-‐for-‐go-‐programs • Profiling and crea^ng call graphs for Go programs with "go tool pprof" | Samuel's Tech Blog • h[p:/ /saml.rilspace.org/profiling-‐and-‐crea^ng-‐call-‐graphs-‐for-‐go-‐ programs-‐with-‐go-‐tool-‐pprof

Slide 8

Slide 8 text

tes?ng.B • ςετͱಉ͡Α͏ͳܗͰϕϯνϚʔΫΛऔΔख๏΋ඪ४Ͱඋ Θ͍ͬͯΔ • Go ͰϕϯνϚʔΫ -‐ Block Rockin’ Codes h[p:/ / jxck.hatenablog.com/entry/20131123/1385189088 • allocճ਺ͳͲ΋෼͔ΔΦϓγϣϯ͋Γ • ϘτϧωοΫ͑͞෼͔͍ͬͯΕ͹ϚΠΫϩϕϯν͸؆୯ʹऔ ΕΔ • ࡉ͔͍࣮૷Ͱ໎͍͕ग़ͨΒ͙͢ϕϯνϚʔΫ

Slide 9

Slide 9 text

The status of the app in Go • run^me͕Ͳ͏ͳ͍ͬͯΔͷ͔ݟ͍ͨ • github:fukata/golang-‐stats-‐api-‐handler • GoαʔόͷϞχλϦϯά -‐ Qiita h[p:/ /qiita.com/cubicdaiya/items/ b90c1ae147a72f4970b2 • net/h[p/pprofΛ࢖͏ͱWebͰpprofͷ݁Ռ͕ݟΕΔ • golang ͷ net/h[p/pprof Λ৮ͬͯΈͨϝϞ -‐ sonots:blog h[p:/ /blog.livedoor.jp/ sonots/archives/39879160.html • Nginx͔Β͜ΕΒͷ݁Ռ͸ࣾ಺͔Β͔͠ݟΕͳ͍Α͏ʹ • σϓϩΠ͞ΕͨόʔδϣϯΛ஌Γ͍ͨ • go build -‐ldﬂags "-‐X main.appVersion `git rev-‐list HEAD -‐n1`"

Slide 10

Slide 10 text

Monitoring • Goͷϓϩηεͷࢮ׆؂ࢹ͸Circus • ࣾ಺Ͱ͸NagiosΛ࢖͍ͬͯΔͷͰNagiosͰҎԼͷ΋ͷΛ؂ࢹ • εςʔλείʔυ200൪Λฦ͚ͩ͢ͷϧʔςΟϯά • Nginxͷstub_statusϓϥάΠϯͰઃఆͨ͠ϧʔςΟϯά • circusctl statusͷ࣮ߦ݁Ռ • golang-‐stats-‐api-‐handlerͷ݁ՌΛmuninͰάϥϑԽ • Τϥʔϩά͸fluentdͰࣾ಺ͷΤϥʔϩάू໿αʔόʔʹૹͬͯ؂ࢹ • fluentd͸ΞϓϦέʔγϣϯ༻ͱΤϥʔϩά༻ʹ2ͭىಈ͍ͯ͠Δ • fluentdࣗମ΋monitor_agentΛ࢖ͬͯNagios͔Β؂ࢹ

Slide 11

Slide 11 text

Cache • ΞΫηεͷ౓ʹmemcachedʹΞΫηεΛͨ͠Β஗্͍ʹCPU࢖༻཰΋্͕ͬͯ͠·ͬͨ • pprof͔Β͸memcached͕஗͍ͱ͍͏͜ͱ͕Α͘෼͔Βͳ͔ͬͨ • ඇಉظʹσʔλΛऔಘ͢ΔͷͰpprofͰ͸͏·͘ग़ͯ͜ͳ͍ • github:walf443/stopwatch • memcached ͷऔಘ݁ՌΛΞϓϦέʔγϣϯଆͰΩϟογϡ͢Δ͜ͱʹ • ௕͗͢Δͱ޿ࠂσʔλͷ൓ө͕஗ΕΔ • github:pmylund/go-‐cache • άϩʔόϧʹcacheͱ͍͏ม਺໊Λ࢖ͬͯ͠·͏ͷͰimport࣌ʹผͷ໊લΛ͚ͭΔͷ ͕͓͢͢Ί

Slide 12

Slide 12 text

Faster

Slide 13

Slide 13 text

CPU throGling • ిྗফඅΛ཈͑ΔͨΊʹෛՙ͕͔͔͍ͬͯͳ͍ؒ͸CPUΛ࠷େप೾਺ΑΓ΋௿͍प೾਺Ͱಈ ͔͢ػೳ • /proc/cpuinfo͔Βݱࡏͷಈ࡞प೾਺Λ֬ೝՄೳ • ޿ࠂ഑৴αʔόʔ͸ৗʹߴ଎ʹಈ࡞ͯ͠ཉ͍͠ • ap^tude install cpufrequ^ls • /etc/default/cpufrequ^lsʹCPUFREQ_OPTIONS=“-‐r”㾑 GOVERNOR=“performance” • ࣌ؒଳʹΑͬͯ͸2ഒҎ্ͷϨεϙϯελΠϜͰฦͤΔΑ͏ʹ • Քಇॳظ͔ΒmuninͷCPU࢖༻཰ͷάϥϑ͕಄ΛଧͬͨΑ͏ʹͳΔ໰୊͕͕͋ͬͨৗʹ࠷େ प೾਺Ͱಈ͔͢͜ͱͰվળ • ޿ࠂαʔόʔ͸ϦΫΤετ͕དྷͨޙʹߴ଎ʹϨεϙϯεΛฦͨ͢Ί࠷େप೾਺·Ͱ্͕ Γʹ͍͘ʁ

Slide 14

Slide 14 text

No content

Slide 15

Slide 15 text

Log output in Go • ࠷ॳ͸Go͔Βstdoutܦ༝ͰϩάΛग़ྗ͠ɺCircus͕ϩάϑΝΠϧԽ͢Δߏ੒ • ॳظ࣮૷ίετ͕཈͑ΒΕΔϝϦοτ • ϐʔΫλΠϜ࣌ͷCircusͷCPU࢖༻཰͕ߴ͘ͳΔσϝϦοτ • Go͔Β௚઀ϑΝΠϧΛॻ͖ࠐΉΑ͏ʹमਖ਼ • ϩάϩʔςʔτΛࣗલͰ࣮૷͢Δ • USR1γάφϧΛૹΒΕͨΒϑΝΠϧΛ։͖௚͢ • logrotateͷpostrotateͰrotateޙʹεΫϦϓτΛ࣮ߦͯ͠USR1γάφϧΛૹΔ • Go੡ͷαʔόʔͰىಈ࣌ʹPIDϑΝΠϧΛ࡞Δ -‐ Qiita h[p:/ /qiita.com/catatsuy/items/ a485066ca9d4115dd213 • Τϥʔϩά͸stderrܦ༝ͷ··ͰCircus͕ϩάϑΝΠϧԽ

Slide 16

Slide 16 text

• GoͰड͚औͬͨγάφϧʹΑͬͯಈ͖Λม͍͑ͨ -‐ Qiita h[p:/ /qiita.com/catatsuy/items/ 3048b66041151425e806

Slide 17

Slide 17 text

Conclusion • GoͰ޿ࠂ഑৴αʔόʔΛ࡞ͬͯӡ༻͍ͯ͠Δ • େ͖ͳτϥϒϧ͸ى͍ͬͯ͜ͳ͍ • ίϯύΠϥݴޠͳͷͰϦϑΝΫλϦϯά΋ָ • ϓϩϑΝΠϦϯά΋औΓ΍͍͢ͷͰύϑΥʔϚϯε͕ඞཁͳ৔໘Ͱ΋࢖͍΍ ͍͢ • Nginx, CircusΛ࢖͏͜ͱͰ࠷খݶͷ࣮૷ͰελʔτͰ͖ͨ • LinuxΧʔωϧͷলిྗػೳͷӨڹΛड͚΍͍͢ͷͰύϑΥʔϚϯε͕ཉ͍͠ ͳΒΦϑʹ • ΞϓϦέʔγϣϯͷঢ়ଶΛݟΔ͜ͱ΋؆୯