Upgrade to Pro — share decks privately, control downloads, hide ads and more …

はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6

tatsuru
PRO
August 31, 2016
11k

はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6

Hatena Engineer Seminar #6 インフラ編での発表スライドです

tatsuru
PRO

August 31, 2016
Tweet

Transcript

  1. ͸ͯͳͷϩάӡ༻
    ͜Ε·Ͱͱ͜Ε͔Β
    id:wtatsuru
    2016/08/31 Hatena Engineer Seminar #6

    View Slide

  2. ࣗݾ঺հ
    • Tatsuru Watabane (id:wtatsuru)
    • ͸ͯͳγεςϜϓϥοτϑΥʔϜ෦௕
    • 2011೥ೖࣾ
    • 2014೥·ͰͷαʔϏε͸͍͍ͩͨશ෦ӡ༻

    View Slide

  3. ຊ೔ͷ಺༰
    • લఏͱͳΔγεςϜߏ੒
    • ͸ͯͳͷϩάͷߏ੒ͱӡ༻Λ঺հ
    • ࠓޙ

    View Slide

  4. લఏͱͳΔγεςϜߏ੒

    View Slide

  5. 3૚ߏ଄
    reverse proxy
    App server
    DB

    View Slide

  6. ѻ͏ϩά
    reverse proxy
    App server
    DB
    access log
    error log
    various logs

    View Slide

  7. ѻ͏ϩά
    • αʔϏεͷΞΫηεϩά
    • ˏϦόʔεϓϩΩγ
    • ΞϓϦέʔγϣϯ͕ు͘ϩά
    • Τϥʔϩάɺߦಈϩά etc.
    reverse proxy
    App server
    DB
    access log
    error log
    various logs

    View Slide

  8. લఏɿଟ͘ͷαʔϏε
    • ৽αʔϏε͔Β10೥Ҏ্ଓ͘αʔϏε·Ͱଟ༷
    • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ
    • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ

    View Slide

  9. ͸ͯͳͷϩάӡ༻

    View Slide

  10. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    syslog server
    storage server
    SSD server
    rsyslog
    aggregator

    View Slide

  11. ͸ͯͳͷϩάӡ༻
    • ΞΫηεϩάͷӡ༻
    • ΞϓϦέʔγϣϯϩάͷӡ༻
    • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

    View Slide

  12. ΞΫηεϩάग़ྗ
    • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ
    • ֤छௐࠪɺૢ࡞ϩάɺetc.
    • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ
    • LTSV (Labeled Tab-separated Values) http://ltsv.org/
    host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700]
    req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http://
    www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)

    View Slide

  13. reverse proxy
    App server
    DB
    access log
    ΞΫηεϩάग़ྗ

    View Slide

  14. ϩάΛอଘ͍ͨ͠
    • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ
    • rsyslog Ͱඈ͹ͯ͠ετϨʔδαʔόʹอଘ
    • ϩʔΧϧ͔ΒS3ʹ΋อଘͯ͠ೋॏԽ
    • ͠͹Β͘͢ΔͱGlacierʹ௜Ή

    View Slide

  15. reverse proxy
    App server
    DB
    access log
    syslog server
    rsyslog
    storage server

    View Slide

  16. ؆୯ʹ෼ੳ͍ͨ͠
    • SSDͷαʔόʹ΋ඈ͹͢
    • SSD͕҆͘ͳͬͨͷͰಋೖ
    • ࡶଟʹ grep ΍෼ੳ
    • ΞΫηεϩά͔ΒϨΠςϯγ෼ੳ
    • 2೥ॻ͍ͨΒSSD͕յΕͨͷͰަ׵

    View Slide

  17. reverse proxy
    App server
    DB
    access log
    syslog server
    storage server
    SSD server
    rsyslog

    View Slide

  18. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠
    • S3 ͔Β Amazon EMR (ElasticMapReduce) ΁
    • LTSVΛಡΈࠐΉσγϦΞϥΠβ 

    github.com/hatena/KeyValuePairsDeserializer
    • SQLͰ؆୯ʹ෼ੳ
    select count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘

    View Slide

  19. reverse proxy
    App server
    DB
    access log
    syslog server
    storage server
    SSD server
    rsyslog

    View Slide

  20. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠
    • ֤छ՝୊
    • ॳظԽ͕஗͍
    • ؀ڥߏங͕೉͍͠
    • ఆظతʹճ͢ͱίετ͕
    • ࢖ΘΕͳ͘ͳ͍ͬͯ͘…

    View Slide

  21. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠
    • EMRஔ͖׵͑ɿGoogle BigQuery Λݕূ
    • LTSV ͸ embulk Ͱલॲཧͯ͠ಡΈࠐΉ
    • ଎͍ɾ·͋·͍͋҆
    • ։ൃऀҎ֎Ͱ΋࢖͍΍͍͢

    View Slide

  22. reverse proxy
    App server
    DB
    access log
    syslog server
    storage server
    SSD server
    rsyslog

    View Slide

  23. ͸ͯͳͷϩάӡ༻
    • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ
    • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β
    • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

    View Slide

  24. ΞϓϦέʔγϣϯͷϩά
    • ૉ๿ͳੲ
    • /var/log/httpd/access.log
    • /var/log/httpd/error.log
    • ֤छͪ͝Ό·ͥ
    • Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍
    • DBʹೖΕΔ

    View Slide

  25. reverse proxy
    App server
    DB

    View Slide

  26. ߏ଄ԽϩάΛग़͢
    • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘
    • ඪ४ग़ྗɾΤϥʔग़ྗͱ͸෼͚Δ
    • ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘
    • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ΋

    View Slide

  27. reverse proxy
    App server
    DB
    structured log

    View Slide

  28. fluentdͰ഑ૹ
    • ֤ϗετͰ in_tail +out_forward Ͱඈ͹͚ͩ͢
    • ͜͜͸΄ͱΜͲ͍͡Βͳ͍
    • தԝͷ aggregator ʹ౉ͯ͠Α͠ͳʹॲཧ
    • ඼࣭͸ϕετΤϑΥʔτ
    • ඞཁͳ৔ॴʹ͸ผ్อূͨ͠࢓૊ΈΛཱͯΔ
    • ͍ͭͰʹϦόʔεϓϩΩγ͔Β΋ૹΓ࢝ΊΔ

    View Slide

  29. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View Slide

  30. ֤ॴʹ഑ૹ
    • S3 ʹੜσʔλΛอଘ
    • Կ͔ίέͯ΋࠶ݱ
    • شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ

    View Slide

  31. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View Slide

  32. ϩάͷ഑ૹ
    • Mackerelʹ౤ߘ
    • ֤छϝτϦοΫՄࢹԽ
    • ϨΠςϯγ
    • ΤϥʔϨʔτ
    • KPI

    View Slide

  33. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View Slide

  34. ϩάͷ഑ૹ
    • Slackʹ௨஌
    • ΤϥʔϩάΛूΊΔ
    • ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……

    View Slide

  35. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View Slide

  36. ϩάͷ഑ૹ
    • Elasticsearch→Kibana ͰՄࢹԽ
    • ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ

    View Slide

  37. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View Slide

  38. ϩάͷ഑ૹ
    • MongoDB capped collection ʹ౤ೖ
    • ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ
    • ʢ࠷ۙ͋·Γ࢖ΘΕ͍ͯͳ͍ʣ

    View Slide

  39. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator

    View Slide

  40. શܠ

    View Slide

  41. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    syslog server
    storage server
    SSD server
    rsyslog
    aggregator

    View Slide

  42. ࠓޙ
    • syslog ͔Βͷ୤٫
    • ։ൃऀҎ֎΋׆༻Ͱ͖Δ࢓૊Έ࡞Γ
    • ߦಈϩάج൫

    View Slide

  43. ຊ೔ͷ಺༰
    • ͸ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λ঺հ͠·ͨ͠
    • ࠓޙ΋஍ಓͳվળΛଓ͚͍͖ͯ·͢

    View Slide

  44. Ҏ্Ͱ͢
    • Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ

    http://hatenacorp.jp/recruit/career/operation-engineer
    • ౦ژɾژ౎ͲͪΒͰ΋

    View Slide

  45. ࣭ٙɾTwitterΑΓ
    • fluentd aggregator ٧·Γ·ͤΜ͔
    • aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝
    ͱʹ fluentd Λཱͯͯอଘ͍ͯ͠·͢
    • ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ
    ͍ͯ͠·͢

    View Slide

  46. ࣭ٙɾTwitterΑΓ
    • ߦಈϩάͰ͸ܽଛ΍੔߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜
    Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢
    • কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ
    ֊Ͱ͢…
    • ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬ
    ͯܽଛ͠ͳ͍࢓૊ΈΛ࡞Γ·͢ɻKafka ͷΑ͏ͳ
    ΩϡʔΛڬΉͳͲ

    View Slide

  47. ࣭ٙɾTwitterΑΓ
    • յΕͨSSD͸Ͳ͜ͷͰ͔͢
    • ॳ୅͸ Crucial M4 Λ࢖͍ͬͯ·ͨ͠

    View Slide