Upgrade to Pro — share decks privately, control downloads, hide ads and more …

はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6

B642bff07fa32e9574c6647d6a83b35f?s=47 tatsuru
August 31, 2016
11k

はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6

Hatena Engineer Seminar #6 インフラ編での発表スライドです

B642bff07fa32e9574c6647d6a83b35f?s=128

tatsuru

August 31, 2016
Tweet

Transcript

  1. ͸ͯͳͷϩάӡ༻ ͜Ε·Ͱͱ͜Ε͔Β id:wtatsuru 2016/08/31 Hatena Engineer Seminar #6

  2. ࣗݾ঺հ • Tatsuru Watabane (id:wtatsuru) • ͸ͯͳγεςϜϓϥοτϑΥʔϜ෦௕ • 2011೥ೖࣾ •

    2014೥·ͰͷαʔϏε͸͍͍ͩͨશ෦ӡ༻
  3. ຊ೔ͷ಺༰ • લఏͱͳΔγεςϜߏ੒ • ͸ͯͳͷϩάͷߏ੒ͱӡ༻Λ঺հ • ࠓޙ

  4. લఏͱͳΔγεςϜߏ੒

  5. 3૚ߏ଄ reverse proxy App server DB

  6. ѻ͏ϩά reverse proxy App server DB access log error log

    various logs
  7. ѻ͏ϩά • αʔϏεͷΞΫηεϩά • ˏϦόʔεϓϩΩγ • ΞϓϦέʔγϣϯ͕ు͘ϩά • Τϥʔϩάɺߦಈϩά etc.

    reverse proxy App server DB access log error log various logs
  8. લఏɿଟ͘ͷαʔϏε • ৽αʔϏε͔Β10೥Ҏ্ଓ͘αʔϏε·Ͱଟ༷ • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ

  9. ͸ͯͳͷϩάӡ༻

  10. reverse proxy App server DB access log error log structured

    log syslog server storage server SSD server rsyslog aggregator
  11. ͸ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ • ΞϓϦέʔγϣϯϩάͷӡ༻ • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

  12. ΞΫηεϩάग़ྗ • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ • ֤छௐࠪɺૢ࡞ϩάɺetc. • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ • LTSV (Labeled

    Tab-separated Values) http://ltsv.org/ host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700] req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http:// www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)
  13. reverse proxy App server DB access log ΞΫηεϩάग़ྗ

  14. ϩάΛอଘ͍ͨ͠ • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ • rsyslog Ͱඈ͹ͯ͠ετϨʔδαʔόʹอଘ • ϩʔΧϧ͔ΒS3ʹ΋อଘͯ͠ೋॏԽ • ͠͹Β͘͢ΔͱGlacierʹ௜Ή

  15. reverse proxy App server DB access log syslog server rsyslog

    storage server
  16. ؆୯ʹ෼ੳ͍ͨ͠ • SSDͷαʔόʹ΋ඈ͹͢ • SSD͕҆͘ͳͬͨͷͰಋೖ • ࡶଟʹ grep ΍෼ੳ •

    ΞΫηεϩά͔ΒϨΠςϯγ෼ੳ • 2೥ॻ͍ͨΒSSD͕յΕͨͷͰަ׵
  17. reverse proxy App server DB access log syslog server storage

    server SSD server rsyslog
  18. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • S3 ͔Β Amazon EMR (ElasticMapReduce) ΁ • LTSVΛಡΈࠐΉσγϦΞϥΠβ

    
 github.com/hatena/KeyValuePairsDeserializer • SQLͰ؆୯ʹ෼ੳ select count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘
  19. reverse proxy App server DB access log syslog server storage

    server SSD server rsyslog
  20. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • ֤छ՝୊ • ॳظԽ͕஗͍ • ؀ڥߏங͕೉͍͠ • ఆظతʹճ͢ͱίετ͕ •

    ࢖ΘΕͳ͘ͳ͍ͬͯ͘…
  21. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • EMRஔ͖׵͑ɿGoogle BigQuery Λݕূ • LTSV ͸ embulk Ͱલॲཧͯ͠ಡΈࠐΉ

    • ଎͍ɾ·͋·͍͋҆ • ։ൃऀҎ֎Ͱ΋࢖͍΍͍͢
  22. reverse proxy App server DB access log syslog server storage

    server SSD server rsyslog
  23. ͸ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

  24. ΞϓϦέʔγϣϯͷϩά • ૉ๿ͳੲ • /var/log/httpd/access.log • /var/log/httpd/error.log • ֤छͪ͝Ό·ͥ •

    Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍ • DBʹೖΕΔ
  25. reverse proxy App server DB

  26. ߏ଄ԽϩάΛग़͢ • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘ • ඪ४ग़ྗɾΤϥʔग़ྗͱ͸෼͚Δ •

    ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘ • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ΋
  27. reverse proxy App server DB structured log

  28. fluentdͰ഑ૹ • ֤ϗετͰ in_tail +out_forward Ͱඈ͹͚ͩ͢ • ͜͜͸΄ͱΜͲ͍͡Βͳ͍ • தԝͷ

    aggregator ʹ౉ͯ͠Α͠ͳʹॲཧ • ඼࣭͸ϕετΤϑΥʔτ • ඞཁͳ৔ॴʹ͸ผ్อূͨ͠࢓૊ΈΛཱͯΔ • ͍ͭͰʹϦόʔεϓϩΩγ͔Β΋ૹΓ࢝ΊΔ
  29. reverse proxy App server DB access log error log structured

    log aggregator
  30. ֤ॴʹ഑ૹ • S3 ʹੜσʔλΛอଘ • Կ͔ίέͯ΋࠶ݱ • شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ

  31. reverse proxy App server DB access log error log structured

    log aggregator
  32. ϩάͷ഑ૹ • Mackerelʹ౤ߘ • ֤छϝτϦοΫՄࢹԽ • ϨΠςϯγ • ΤϥʔϨʔτ •

    KPI
  33. reverse proxy App server DB access log error log structured

    log aggregator
  34. ϩάͷ഑ૹ • Slackʹ௨஌ • ΤϥʔϩάΛूΊΔ • ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……

  35. reverse proxy App server DB access log error log structured

    log aggregator
  36. ϩάͷ഑ૹ • Elasticsearch→Kibana ͰՄࢹԽ • ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ

  37. reverse proxy App server DB access log error log structured

    log aggregator
  38. ϩάͷ഑ૹ • MongoDB capped collection ʹ౤ೖ • ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ • ʢ࠷ۙ͋·Γ࢖ΘΕ͍ͯͳ͍ʣ

  39. reverse proxy App server DB access log error log structured

    log aggregator
  40. શܠ

  41. reverse proxy App server DB access log error log structured

    log syslog server storage server SSD server rsyslog aggregator
  42. ࠓޙ • syslog ͔Βͷ୤٫ • ։ൃऀҎ֎΋׆༻Ͱ͖Δ࢓૊Έ࡞Γ • ߦಈϩάج൫

  43. ຊ೔ͷ಺༰ • ͸ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λ঺հ͠·ͨ͠ • ࠓޙ΋஍ಓͳվળΛଓ͚͍͖ͯ·͢

  44. Ҏ্Ͱ͢ • Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ
 http://hatenacorp.jp/recruit/career/operation-engineer • ౦ژɾژ౎ͲͪΒͰ΋

  45. ࣭ٙɾTwitterΑΓ • fluentd aggregator ٧·Γ·ͤΜ͔ • aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝ ͱʹ fluentd

    Λཱͯͯอଘ͍ͯ͠·͢ • ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ ͍ͯ͠·͢
  46. ࣭ٙɾTwitterΑΓ • ߦಈϩάͰ͸ܽଛ΍੔߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜ Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢ • কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ ֊Ͱ͢… • ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬ ͯܽଛ͠ͳ͍࢓૊ΈΛ࡞Γ·͢ɻKafka

    ͷΑ͏ͳ ΩϡʔΛڬΉͳͲ
  47. ࣭ٙɾTwitterΑΓ • յΕͨSSD͸Ͳ͜ͷͰ͔͢ • ॳ୅͸ Crucial M4 Λ࢖͍ͬͯ·ͨ͠