はてなのログ運用これまでとこれから / Hatena Engineer Seminar #6

by https://speakerdeck.com/tatsuru

Slide 1

Slide 1 text

͸ͯͳͷϩάӡ༻ ͜Ε·Ͱͱ͜Ε͔Β id:wtatsuru 2016/08/31 Hatena Engineer Seminar #6

Slide 2

Slide 2 text

ࣗݾ঺հ • Tatsuru Watabane (id:wtatsuru) • ͸ͯͳγεςϜϓϥοτϑΥʔϜ෦௕ • 2011೥ೖࣾ • 2014೥·ͰͷαʔϏε͸͍͍ͩͨશ෦ӡ༻

Slide 3

Slide 3 text

ຊ೔ͷ಺༰ • લఏͱͳΔγεςϜߏ੒ • ͸ͯͳͷϩάͷߏ੒ͱӡ༻Λ঺հ • ࠓޙ

Slide 4

Slide 4 text

લఏͱͳΔγεςϜߏ੒

Slide 5

Slide 5 text

3૚ߏ଄ reverse proxy App server DB

Slide 6

Slide 6 text

ѻ͏ϩά reverse proxy App server DB access log error log various logs

Slide 7

Slide 7 text

ѻ͏ϩά • αʔϏεͷΞΫηεϩά • ˏϦόʔεϓϩΩγ • ΞϓϦέʔγϣϯ͕ు͘ϩά • Τϥʔϩάɺߦಈϩά etc. reverse proxy App server DB access log error log various logs

Slide 8

Slide 8 text

લఏɿଟ͘ͷαʔϏε • ৽αʔϏε͔Β10೥Ҏ্ଓ͘αʔϏε·Ͱଟ༷ • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ

Slide 9

Slide 9 text

͸ͯͳͷϩάӡ༻

Slide 10

Slide 10 text

reverse proxy App server DB access log error log structured log syslog server storage server SSD server rsyslog aggregator

Slide 11

Slide 11 text

͸ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ • ΞϓϦέʔγϣϯϩάͷӡ༻ • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

Slide 12

Slide 12 text

ΞΫηεϩάग़ྗ • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ • ֤छௐࠪɺૢ࡞ϩάɺetc. • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ • LTSV (Labeled Tab-separated Values) http://ltsv.org/ host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700] req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http:// www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)

Slide 13

Slide 13 text

reverse proxy App server DB access log ΞΫηεϩάग़ྗ

Slide 14

Slide 14 text

ϩάΛอଘ͍ͨ͠ • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ • rsyslog Ͱඈ͹ͯ͠ετϨʔδαʔόʹอଘ • ϩʔΧϧ͔ΒS3ʹ΋อଘͯ͠ೋॏԽ • ͠͹Β͘͢ΔͱGlacierʹ௜Ή

Slide 15

Slide 15 text

reverse proxy App server DB access log syslog server rsyslog storage server

Slide 16

Slide 16 text

؆୯ʹ෼ੳ͍ͨ͠ • SSDͷαʔόʹ΋ඈ͹͢ • SSD͕҆͘ͳͬͨͷͰಋೖ • ࡶଟʹ grep ΍෼ੳ • ΞΫηεϩά͔ΒϨΠςϯγ෼ੳ • 2೥ॻ͍ͨΒSSD͕յΕͨͷͰަ׵

Slide 17

Slide 17 text

reverse proxy App server DB access log syslog server storage server SSD server rsyslog

Slide 18

Slide 18 text

΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • S3 ͔Β Amazon EMR (ElasticMapReduce) ΁ • LTSVΛಡΈࠐΉσγϦΞϥΠβ   github.com/hatena/KeyValuePairsDeserializer • SQLͰ؆୯ʹ෼ੳ select count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘

Slide 19

Slide 19 text

reverse proxy App server DB access log syslog server storage server SSD server rsyslog

Slide 20

Slide 20 text

΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • ֤छ՝୊ • ॳظԽ͕஗͍ • ؀ڥߏங͕೉͍͠ • ఆظతʹճ͢ͱίετ͕ • ࢖ΘΕͳ͘ͳ͍ͬͯ͘…

Slide 21

Slide 21 text

΋ͬͱ؆୯ʹ෼ੳ͍ͨ͠ • EMRஔ͖׵͑ɿGoogle BigQuery Λݕূ • LTSV ͸ embulk Ͱલॲཧͯ͠ಡΈࠐΉ • ଎͍ɾ·͋·͍͋҆ • ։ൃऀҎ֎Ͱ΋࢖͍΍͍͢

Slide 22

Slide 22 text

reverse proxy App server DB access log syslog server storage server SSD server rsyslog

Slide 23

Slide 23 text

͸ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β • ߏஙͷྺ࢙ͱͱ΋ʹΏΔ͘঺հ͠·͢

Slide 24

Slide 24 text

ΞϓϦέʔγϣϯͷϩά • ૉ๿ͳੲ • /var/log/httpd/access.log • /var/log/httpd/error.log • ֤छͪ͝Ό·ͥ • Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍ • DBʹೖΕΔ

Slide 25

Slide 25 text

reverse proxy App server DB

Slide 26

Slide 26 text

ߏ଄ԽϩάΛग़͢ • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘ • ඪ४ग़ྗɾΤϥʔग़ྗͱ͸෼͚Δ • ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘ • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ΋

Slide 27

Slide 27 text

reverse proxy App server DB structured log

Slide 28

Slide 28 text

ﬂuentdͰ഑ૹ • ֤ϗετͰ in_tail +out_forward Ͱඈ͹͚ͩ͢ • ͜͜͸΄ͱΜͲ͍͡Βͳ͍ • தԝͷ aggregator ʹ౉ͯ͠Α͠ͳʹॲཧ • ඼࣭͸ϕετΤϑΥʔτ • ඞཁͳ৔ॴʹ͸ผ్อূͨ͠࢓૊ΈΛཱͯΔ • ͍ͭͰʹϦόʔεϓϩΩγ͔Β΋ૹΓ࢝ΊΔ