はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6

Hatena Engineer Seminar #6 インフラ編での発表スライドです

Transcript

1. ͸ͯͳͷϩάӡ༻
   ͜Ε·Ͱͱ͜Ε͔Β
   id:wtatsuru
   2016/08/31 Hatena Engineer Seminar #6
   

   View Slide

2. ࣗݾ঺հ
   • Tatsuru Watabane (id:wtatsuru)
   • ͸ͯͳγεςϜϓϥοτϑΥʔϜ෦௕
   • 2011೥ೖࣾ
   • 2014೥·ͰͷαʔϏε͸͍͍ͩͨશ෦ӡ༻
   

   View Slide

3. ຊ೔ͷ಺༰
   • લఏͱͳΔγεςϜߏ੒
   • ͸ͯͳͷϩάͷߏ੒ͱӡ༻Λ঺հ
   • ࠓޙ
   

   View Slide

4. લఏͱͳΔγεςϜߏ੒
   

   View Slide

5. 3૚ߏ଄
   reverse proxy
   App server
   DB
   

   View Slide

6. ѻ͏ϩά
   reverse proxy
   App server
   DB
   access log
   error log
   various logs
   

   View Slide

7. ѻ͏ϩά
   • αʔϏεͷΞΫηεϩά
   • ˏϦόʔεϓϩΩγ
   • ΞϓϦέʔγϣϯ͕ు͘ϩά
   • Τϥʔϩάɺߦಈϩά etc.
   reverse proxy
   App server
   DB
   access log
   error log
   various logs
   

   View Slide

8. લఏɿଟ͘ͷαʔϏε
   • ৽αʔϏε͔Β10೥Ҏ্ଓ͘αʔϏε·Ͱଟ༷
   • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ
   • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ
   

   View Slide

9. ͸ͯͳͷϩάӡ༻
   

   View Slide

10. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    syslog server
    storage server
    SSD server
    rsyslog
    aggregator
    

    View Slide

11. ͸ͯͳͷϩάӡ༻
    • ΞΫηεϩάͷӡ༻
    • ΞϓϦέʔγϣϯϩάͷӡ༻
    • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢
    

    View Slide

12. ΞΫηεϩάग़ྗ
    • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ
    • ֤छௐࠪɺૢ࡞ϩάɺetc.
    • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ
    • LTSV (Labeled Tab-separated Values) http://ltsv.org/
    host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700]
    req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http://
    www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)
    

    View Slide

13. reverse proxy
    App server
    DB
    access log
    ΞΫηεϩάग़ྗ
    

    View Slide

14. ϩάΛอଘ͍ͨ͠
    • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ
    • rsyslog Ͱඈ͹ͯ͠ετϨʔδαʔόʹอଘ
    • ϩʔΧϧ͔ΒS3ʹ΋อଘͯ͠ೋॏԽ
    • ͠͹Β͘͢ΔͱGlacierʹ௜Ή
    

    View Slide

15. reverse proxy
    App server
    DB
    access log
    syslog server
    rsyslog
    storage server
    

    View Slide

16. ؆୯ʹ෼ੳ͍ͨ͠
    • SSDͷαʔόʹ΋ඈ͹͢
    • SSD͕҆͘ͳͬͨͷͰಋೖ
    • ࡶଟʹ grep ΍෼ੳ
    • ΞΫηεϩά͔ΒϨΠςϯγ෼ੳ
    • 2೥ॻ͍ͨΒSSD͕յΕͨͷͰަ׵
    

    View Slide

17. reverse proxy
    App server
    DB
    access log
    syslog server
    storage server
    SSD server
    rsyslog
    

    View Slide

18. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠
    • S3 ͔Β Amazon EMR (ElasticMapReduce) ΁
    • LTSVΛಡΈࠐΉσγϦΞϥΠβ 

    github.com/hatena/KeyValuePairsDeserializer
    • SQLͰ؆୯ʹ෼ੳ
    select count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘
    

    View Slide

19. reverse proxy
    App server
    DB
    access log
    syslog server
    storage server
    SSD server
    rsyslog
    

    View Slide

20. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠
    • ֤छ՝୊
    • ॳظԽ͕஗͍
    • ؀ڥߏங͕೉͍͠
    • ఆظతʹճ͢ͱίετ͕
    • ࢖ΘΕͳ͘ͳ͍ͬͯ͘…
    

    View Slide

21. ΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠
    • EMRஔ͖׵͑ɿGoogle BigQuery Λݕূ
    • LTSV ͸ embulk Ͱલॲཧͯ͠ಡΈࠐΉ
    • ଎͍ɾ·͋·͍͋҆
    • ։ൃऀҎ֎Ͱ΋࢖͍΍͍͢
    

    View Slide

22. reverse proxy
    App server
    DB
    access log
    syslog server
    storage server
    SSD server
    rsyslog
    

    View Slide

23. ͸ͯͳͷϩάӡ༻
    • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ
    • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β
    • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢
    

    View Slide

24. ΞϓϦέʔγϣϯͷϩά
    • ૉ๿ͳੲ
    • /var/log/httpd/access.log
    • /var/log/httpd/error.log
    • ֤छͪ͝Ό·ͥ
    • Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍
    • DBʹೖΕΔ
    

    View Slide

25. reverse proxy
    App server
    DB
    

    View Slide

26. ߏ଄ԽϩάΛग़͢
    • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘
    • ඪ४ग़ྗɾΤϥʔग़ྗͱ͸෼͚Δ
    • ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘
    • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ΋
    

    View Slide

27. reverse proxy
    App server
    DB
    structured log
    

    View Slide

28. fluentdͰ഑ૹ
    • ֤ϗετͰ in_tail +out_forward Ͱඈ͹͚ͩ͢
    • ͜͜͸΄ͱΜͲ͍͡Βͳ͍
    • தԝͷ aggregator ʹ౉ͯ͠Α͠ͳʹॲཧ
    • ඼࣭͸ϕετΤϑΥʔτ
    • ඞཁͳ৔ॴʹ͸ผ్อূͨ͠࢓૊ΈΛཱͯΔ
    • ͍ͭͰʹϦόʔεϓϩΩγ͔Β΋ૹΓ࢝ΊΔ
    

    View Slide

29. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator
    

    View Slide

30. ֤ॴʹ഑ૹ
    • S3 ʹੜσʔλΛอଘ
    • Կ͔ίέͯ΋࠶ݱ
    • شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ
    

    View Slide

31. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator
    

    View Slide

32. ϩάͷ഑ૹ
    • Mackerelʹ౤ߘ
    • ֤छϝτϦοΫՄࢹԽ
    • ϨΠςϯγ
    • ΤϥʔϨʔτ
    • KPI
    

    View Slide

33. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator
    

    View Slide

34. ϩάͷ഑ૹ
    • Slackʹ௨஌
    • ΤϥʔϩάΛूΊΔ
    • ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……
    

    View Slide

35. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator
    

    View Slide

36. ϩάͷ഑ૹ
    • Elasticsearch→Kibana ͰՄࢹԽ
    • ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ
    

    View Slide

37. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator
    

    View Slide

38. ϩάͷ഑ૹ
    • MongoDB capped collection ʹ౤ೖ
    • ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ
    • ʢ࠷ۙ͋·Γ࢖ΘΕ͍ͯͳ͍ʣ
    

    View Slide

39. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    aggregator
    

    View Slide

40. શܠ
    

    View Slide

41. reverse proxy
    App server
    DB
    access log
    error log
    structured log
    syslog server
    storage server
    SSD server
    rsyslog
    aggregator
    

    View Slide

42. ࠓޙ
    • syslog ͔Βͷ୤٫
    • ։ൃऀҎ֎΋׆༻Ͱ͖Δ࢓૊Έ࡞Γ
    • ߦಈϩάج൫
    

    View Slide

43. ຊ೔ͷ಺༰
    • ͸ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λ঺հ͠·ͨ͠
    • ࠓޙ΋஍ಓͳվળΛଓ͚͍͖ͯ·͢
    

    View Slide

44. Ҏ্Ͱ͢
    • Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ

    http://hatenacorp.jp/recruit/career/operation-engineer
    • ౦ژɾژ౎ͲͪΒͰ΋
    

    View Slide

45. ࣭ٙɾTwitterΑΓ
    • fluentd aggregator ٧·Γ·ͤΜ͔
    • aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝
    ͱʹ fluentd Λཱͯͯอଘ͍ͯ͠·͢
    • ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ
    ͍ͯ͠·͢
    

    View Slide

46. ࣭ٙɾTwitterΑΓ
    • ߦಈϩάͰ͸ܽଛ΍੔߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜
    Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢
    • কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ
    ֊Ͱ͢…
    • ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬ
    ͯܽଛ͠ͳ͍࢓૊ΈΛ࡞Γ·͢ɻKafka ͷΑ͏ͳ
    ΩϡʔΛڬΉͳͲ
    

    View Slide

47. ࣭ٙɾTwitterΑΓ
    • յΕͨSSD͸Ͳ͜ͷͰ͔͢
    • ॳ୅͸ Crucial M4 Λ࢖͍ͬͯ·ͨ͠
    

    View Slide