$30 off During Our Annual Pro Sale. View Details »

ピクシブ社内広告サーバーでのGoの開発・運用 #gocon /p_ads_server_gocon2015

ピクシブ社内広告サーバーでのGoの開発・運用 #gocon /p_ads_server_gocon2015

ピクシブ社内広告サーバーでのGoの開発・運用

Kaneko Tatsuya

June 21, 2015
Tweet

More Decks by Kaneko Tatsuya

Other Decks in Technology

Transcript

  1. ϐΫγϒࣾ಺޿ࠂαʔόʔͰͷ  
    Goͷ։ൃɾӡ༻
    ϐΫγϒ  Πϯϑϥ  catatsuy

    View Slide

  2. About  me
    • catatsuyͱ͍͏IDͰΠϯλʔωοτ্Ͱ׆ಈ  
    • ΧλπΠͱ͔ͬͯݺ͹Ε͍ͯΔ  
    • ϐΫγϒגࣜձࣾͰΠϯϑϥΛ୲౰  
    • ޿ࠂαʔόʔ͕ଟ͍͕ଞʹ΋৭ʑ  
    • एखΠϯϑϥͱͯ͠ͷ࿩ʹ͍ͭͯJTF2015Ͱ࠲ஊձܗࣜͰ࿩͢༧ఆ  
    • ৽ॻͰʰpixivΤϯδχΞ͕ڭ͑Δϓϩάϥϛϯάೖ໳ʱͱ͍͏ຊΛग़͠·ͨ͠  
    • ϐΫγϒࣾ಺ͷඇΤϯδχΞ޲͚ʹ࡞ͬͨݚम಺༰ͷॻ੶Խ  
    • ࠓճ͸ࣾ಺ͷ޿ࠂαʔόʔͰͷGoͷࣄྫΛ঺հ

    View Slide

  3. DNS
    ޿ࠂ഑৴αʔόʔ
    Admin  
    MySQL
    qVFOUE
    NGINX
    IMAGE  
    CACHE
    LTLMPH NFNDBDIFE
    Ad  data
    "QQMJDBUJPO (PMBOH

    Impressions  
    Clicks
    $JSDVT
    Ad  data
    Aggregate  results
    Round  Robin
    6/*9%PNBJO4PDLFU
    'JMFEFTDSJQUPS
    6/*9%PNBJO4PDLFU
    type  tail

    View Slide

  4. Nginx  and  Go
    • ϢʔβʔͷϦΫΤετΛ௚઀Go͕ड͚ΔͷͰ͸ͳ͘NginxͰड͚͍ͨ  
    • ϢʔβʔͷϦΫΤετΛόοϑΝϦϯάͯ͘͠ΕΔͷͰίωΫγϣϯͷอ࣋ͳͲ
    ໘౗ͳ͜ͱΛߟ͑Δඞཁ͕ͳ͘ͳΔ  
    • GoͰ͸௿஗ԆͰϨεϙϯεΛฦ͢͜ͱ͚ͩΛߟ͑Ε͹Α͍  
    • ΞΫηε੍ޚͳͲ΋؆୯  
    • ΩϟογϡͷػߏͳͲ͕؆୯ʹ࢖͑Δ  
    • GoͰtcpΛlistenͯ͠ىಈͨ͠αʔόʔʹରͯ͠proxy_pass͢Δͱ஗͔ͬͨ  
    • ύέοτͷαΠζ͕খ͍͞ͷͰtcpͷ3way  handshakeͷίετ͕ߴ͔ͬͨ  
    • UNIX  domain  socketΛlistenͯ͠αʔόʔΛىಈ͍ͨ͠

    View Slide

  5. Circus
    • Mozilla͕࡞ͬͨPython੡ͷϓϩηε؅ཧπʔϧ  
    • Circusࣗମ͕ιέοτϑΝΠϧΛ࣋ͬͯɼΞϓϦέʔγϣϯ͸ϑΝΠϧ
    σΟεΫϦϓλΛListenͤͯ͞ىಈ͢Δ  
    • σϓϩΠ࣌ʹ৽͍͠ϓϩηε͕ىಈ͔ͯ͠Βݹ͍ϓϩηεΛࡴ͢  
    • μ΢ϯλΠϜͳ͠ͰΞϓϦέʔγϣϯΛೖΕସ͑Δ  
    • circusctlͱ͍͏ίϚϯυπʔϧܦ༝Ͱૢ࡞͢Δ  
    • σϓϩΠ͸όΠφϦΛrsyncͯ͠circusctl  reload͢Δ͚ͩʹ  
    • stdout/stderrͷϩάϑΝΠϧԽͱϩάϩʔςʔτ΋ઃఆ࣍ୈͰՄೳ

    View Slide

  6. UNIX  domain  socket
    • ಉҰϚγϯ্ͳΒ਌ࢠؔ܎͕ແͯ͘΋Ͱ͖Δϓϩηεؒ௨৴  
    • memcachedͱͷ௨৴ʹ΋࢖༻͢Δͱߴ଎͔ͭ҆ఆతʹಈ࡞ͤ͞Δ
    ͜ͱ͕Ͱ͖Δ  
    • CircusΛ࢖͏ͳΒιέοτϑΝΠϧ͸Circus͕࡞੒͢ΔͷͰGoଆͰ͸
    ϑΝΠϧσΟεΫϦϓλΛListen͢ΔαʔόʔΛ࣮૷͢Δඞཁ͕͋Δ  
    • ։ൃΛ༰қʹ͢ΔͨΊʹtcpͰ΋ListenͰ͖ΔΑ͏ʹ͍ͨ͠  
    • net.ListenerΛ༻ҙ͢Ε͹෼ذΛগͳ͘Ͱ͖Δ  
    • net.Listen("unix",  path)  /  net.ListenTCP  /  net.FileListener

    View Slide

  7. Profiling
    • ຊ൪౤ೖલʹͲ͜Ͱ͕͔͔࣌ؒΔͷ͔ݟ͓͖͍ͯͨ  
    • github:davecheney/profile  
    • ͕͔͔࣌ؒΔؔ਺ͳͲΛՄࢹԽ͢ΔPDFϑΝΠϧ΋ग़ྗͰ͖Δ  
    • Introducing  profile,  super  simple  profiling  for  Go  programs  |  Dave  Cheney  
    • h[p:/
    /dave.cheney.net/2013/07/07/introducing-­‐profile-­‐super-­‐simple-­‐
    profiling-­‐for-­‐go-­‐programs  
    • Profiling  and  crea^ng  call  graphs  for  Go  programs  with  "go  tool  pprof"  |  
    Samuel's  Tech  Blog  
    • h[p:/
    /saml.rilspace.org/profiling-­‐and-­‐crea^ng-­‐call-­‐graphs-­‐for-­‐go-­‐
    programs-­‐with-­‐go-­‐tool-­‐pprof

    View Slide

  8. tes?ng.B
    • ςετͱಉ͡Α͏ͳܗͰϕϯνϚʔΫΛऔΔख๏΋ඪ४Ͱඋ
    Θ͍ͬͯΔ  
    • Go  ͰϕϯνϚʔΫ  -­‐  Block  Rockin’  Codes  h[p:/
    /
    jxck.hatenablog.com/entry/20131123/1385189088  
    • allocճ਺ͳͲ΋෼͔ΔΦϓγϣϯ͋Γ  
    • ϘτϧωοΫ͑͞෼͔͍ͬͯΕ͹ϚΠΫϩϕϯν͸؆୯ʹऔ
    ΕΔ  
    • ࡉ͔͍࣮૷Ͱ໎͍͕ग़ͨΒ͙͢ϕϯνϚʔΫ

    View Slide

  9. The  status  of  the  app  in  Go
    • run^me͕Ͳ͏ͳ͍ͬͯΔͷ͔ݟ͍ͨ  
    • github:fukata/golang-­‐stats-­‐api-­‐handler  
    • GoαʔόͷϞχλϦϯά  -­‐  Qiita  h[p:/
    /qiita.com/cubicdaiya/items/
    b90c1ae147a72f4970b2  
    • net/h[p/pprofΛ࢖͏ͱWebͰpprofͷ݁Ռ͕ݟΕΔ  
    • golang  ͷ  net/h[p/pprof  Λ৮ͬͯΈͨϝϞ  -­‐  sonots:blog  h[p:/
    /blog.livedoor.jp/
    sonots/archives/39879160.html  
    • Nginx͔Β͜ΕΒͷ݁Ռ͸ࣾ಺͔Β͔͠ݟΕͳ͍Α͏ʹ  
    • σϓϩΠ͞ΕͨόʔδϣϯΛ஌Γ͍ͨ  
    • go  build  -­‐ldflags  "-­‐X  main.appVersion  `git  rev-­‐list  HEAD  -­‐n1`"

    View Slide

  10. Monitoring
    • Goͷϓϩηεͷࢮ׆؂ࢹ͸Circus  
    • ࣾ಺Ͱ͸NagiosΛ࢖͍ͬͯΔͷͰNagiosͰҎԼͷ΋ͷΛ؂ࢹ  
    • εςʔλείʔυ200൪Λฦ͚ͩ͢ͷϧʔςΟϯά  
    • Nginxͷstub_statusϓϥάΠϯͰઃఆͨ͠ϧʔςΟϯά  
    • circusctl  statusͷ࣮ߦ݁Ռ  
    • golang-­‐stats-­‐api-­‐handlerͷ݁ՌΛmuninͰάϥϑԽ  
    • Τϥʔϩά͸fluentdͰࣾ಺ͷΤϥʔϩάू໿αʔόʔʹૹͬͯ؂ࢹ  
    • fluentd͸ΞϓϦέʔγϣϯ༻ͱΤϥʔϩά༻ʹ2ͭىಈ͍ͯ͠Δ  
    • fluentdࣗମ΋monitor_agentΛ࢖ͬͯNagios͔Β؂ࢹ

    View Slide

  11. Cache
    • ΞΫηεͷ౓ʹmemcachedʹΞΫηεΛͨ͠Β஗্͍ʹCPU࢖༻཰΋্͕ͬͯ͠·ͬͨ  
    • pprof͔Β͸memcached͕஗͍ͱ͍͏͜ͱ͕Α͘෼͔Βͳ͔ͬͨ  
    • ඇಉظʹσʔλΛऔಘ͢ΔͷͰpprofͰ͸͏·͘ग़ͯ͜ͳ͍  
    • github:walf443/stopwatch  
    • memcached  ͷऔಘ݁ՌΛΞϓϦέʔγϣϯଆͰΩϟογϡ͢Δ͜ͱʹ  
    • ௕͗͢Δͱ޿ࠂσʔλͷ൓ө͕஗ΕΔ  
    • github:pmylund/go-­‐cache  
    • άϩʔόϧʹcacheͱ͍͏ม਺໊Λ࢖ͬͯ͠·͏ͷͰimport࣌ʹผͷ໊લΛ͚ͭΔͷ
    ͕͓͢͢Ί

    View Slide

  12. Faster

    View Slide

  13. CPU  throGling
    • ిྗফඅΛ཈͑ΔͨΊʹෛՙ͕͔͔͍ͬͯͳ͍ؒ͸CPUΛ࠷େप೾਺ΑΓ΋௿͍प೾਺Ͱಈ
    ͔͢ػೳ  
    • /proc/cpuinfo͔Βݱࡏͷಈ࡞प೾਺Λ֬ೝՄೳ  
    • ޿ࠂ഑৴αʔόʔ͸ৗʹߴ଎ʹಈ࡞ͯ͠ཉ͍͠  
    • ap^tude  install  cpufrequ^ls  
    • /etc/default/cpufrequ^lsʹCPUFREQ_OPTIONS=“-­‐r”㾑  GOVERNOR=“performance”  
    • ࣌ؒଳʹΑͬͯ͸2ഒҎ্ͷϨεϙϯελΠϜͰฦͤΔΑ͏ʹ  
    • Քಇॳظ͔ΒmuninͷCPU࢖༻཰ͷάϥϑ͕಄ΛଧͬͨΑ͏ʹͳΔ໰୊͕͕͋ͬͨৗʹ࠷େ
    प೾਺Ͱಈ͔͢͜ͱͰվળ  
    • ޿ࠂαʔόʔ͸ϦΫΤετ͕དྷͨޙʹߴ଎ʹϨεϙϯεΛฦͨ͢Ί࠷େप೾਺·Ͱ্͕
    Γʹ͍͘ʁ

    View Slide

  14. View Slide

  15. Log  output  in  Go
    • ࠷ॳ͸Go͔Βstdoutܦ༝ͰϩάΛग़ྗ͠ɺCircus͕ϩάϑΝΠϧԽ͢Δߏ੒  
    • ॳظ࣮૷ίετ͕཈͑ΒΕΔϝϦοτ  
    • ϐʔΫλΠϜ࣌ͷCircusͷCPU࢖༻཰͕ߴ͘ͳΔσϝϦοτ  
    • Go͔Β௚઀ϑΝΠϧΛॻ͖ࠐΉΑ͏ʹमਖ਼  
    • ϩάϩʔςʔτΛࣗલͰ࣮૷͢Δ  
    • USR1γάφϧΛૹΒΕͨΒϑΝΠϧΛ։͖௚͢  
    • logrotateͷpostrotateͰrotateޙʹεΫϦϓτΛ࣮ߦͯ͠USR1γάφϧΛૹΔ  
    • Go੡ͷαʔόʔͰىಈ࣌ʹPIDϑΝΠϧΛ࡞Δ  -­‐  Qiita  h[p:/
    /qiita.com/catatsuy/items/
    a485066ca9d4115dd213  
    • Τϥʔϩά͸stderrܦ༝ͷ··ͰCircus͕ϩάϑΝΠϧԽ

    View Slide

  16. • GoͰड͚औͬͨγάφϧʹΑͬͯಈ͖Λม͍͑ͨ  -­‐  Qiita  h[p:/
    /qiita.com/catatsuy/items/
    3048b66041151425e806

    View Slide

  17. Conclusion
    • GoͰ޿ࠂ഑৴αʔόʔΛ࡞ͬͯӡ༻͍ͯ͠Δ  
    • େ͖ͳτϥϒϧ͸ى͍ͬͯ͜ͳ͍  
    • ίϯύΠϥݴޠͳͷͰϦϑΝΫλϦϯά΋ָ  
    • ϓϩϑΝΠϦϯά΋औΓ΍͍͢ͷͰύϑΥʔϚϯε͕ඞཁͳ৔໘Ͱ΋࢖͍΍
    ͍͢  
    • Nginx,  CircusΛ࢖͏͜ͱͰ࠷খݶͷ࣮૷ͰελʔτͰ͖ͨ  
    • LinuxΧʔωϧͷলిྗػೳͷӨڹΛड͚΍͍͢ͷͰύϑΥʔϚϯε͕ཉ͍͠
    ͳΒΦϑʹ  
    • ΞϓϦέʔγϣϯͷঢ়ଶΛݟΔ͜ͱ΋؆୯

    View Slide