Hatena Engineer Seminar #6 インフラ編での発表スライドです
ͯͳͷϩάӡ༻͜Ε·Ͱͱ͜Ε͔Βid:wtatsuru2016/08/31 Hatena Engineer Seminar #6
View Slide
ࣗݾհ• Tatsuru Watabane (id:wtatsuru)• ͯͳγεςϜϓϥοτϑΥʔϜ෦• 2011ೖࣾ• 2014·ͰͷαʔϏε͍͍ͩͨશ෦ӡ༻
ຊͷ༰• લఏͱͳΔγεςϜߏ• ͯͳͷϩάͷߏͱӡ༻Λհ• ࠓޙ
લఏͱͳΔγεςϜߏ
3ߏreverse proxyApp serverDB
ѻ͏ϩάreverse proxyApp serverDBaccess logerror logvarious logs
ѻ͏ϩά• αʔϏεͷΞΫηεϩά• ˏϦόʔεϓϩΩγ• ΞϓϦέʔγϣϯ͕ు͘ϩά• Τϥʔϩάɺߦಈϩά etc.reverse proxyApp serverDBaccess logerror logvarious logs
લఏɿଟ͘ͷαʔϏε• ৽αʔϏε͔Β10Ҏ্ଓ͘αʔϏε·Ͱଟ༷• ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ• ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ
ͯͳͷϩάӡ༻
reverse proxyApp serverDBaccess logerror logstructured logsyslog serverstorage serverSSD serverrsyslogaggregator
ͯͳͷϩάӡ༻• ΞΫηεϩάͷӡ༻• ΞϓϦέʔγϣϯϩάͷӡ༻• ߏஙͷྺ࢙ͱͱʹΏΔ͘հ͠·͢
ΞΫηεϩάग़ྗ• ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ• ֤छௐࠪɺૢ࡞ϩάɺetc.• ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ• LTSV (Labeled Tab-separated Values) http://ltsv.org/host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700]req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http://www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)
reverse proxyApp serverDBaccess logΞΫηεϩάग़ྗ
ϩάΛอଘ͍ͨ͠• ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ• rsyslog Ͱඈͯ͠ετϨʔδαʔόʹอଘ• ϩʔΧϧ͔ΒS3ʹอଘͯ͠ೋॏԽ• ͠Β͘͢ΔͱGlacierʹΉ
reverse proxyApp serverDBaccess logsyslog serverrsyslogstorage server
؆୯ʹੳ͍ͨ͠• SSDͷαʔόʹඈ͢• SSD͕҆͘ͳͬͨͷͰಋೖ• ࡶଟʹ grep ੳ• ΞΫηεϩά͔ΒϨΠςϯγੳ• 2ॻ͍ͨΒSSD͕յΕͨͷͰަ
reverse proxyApp serverDBaccess logsyslog serverstorage serverSSD serverrsyslog
ͬͱ؆୯ʹੳ͍ͨ͠• S3 ͔Β Amazon EMR (ElasticMapReduce) • LTSVΛಡΈࠐΉσγϦΞϥΠβ github.com/hatena/KeyValuePairsDeserializer• SQLͰ؆୯ʹੳselect count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘
ͬͱ؆୯ʹੳ͍ͨ͠• ֤छ՝• ॳظԽ͕͍• ڥߏங͕͍͠• ఆظతʹճ͢ͱίετ͕• ΘΕͳ͘ͳ͍ͬͯ͘…
ͬͱ؆୯ʹੳ͍ͨ͠• EMRஔ͖͑ɿGoogle BigQuery Λݕূ• LTSV embulk Ͱલॲཧͯ͠ಡΈࠐΉ• ͍ɾ·͋·͍͋҆• ։ൃऀҎ֎Ͱ͍͍͢
ͯͳͷϩάӡ༻• ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ• ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β• ߏஙͷྺ࢙ͱͱʹΏΔ͘հ͠·͢
ΞϓϦέʔγϣϯͷϩά• ૉͳੲ• /var/log/httpd/access.log• /var/log/httpd/error.log• ֤छͪ͝Ό·ͥ• Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍• DBʹೖΕΔ
reverse proxyApp serverDB
ߏԽϩάΛग़͢• શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘• ඪ४ग़ྗɾΤϥʔग़ྗͱ͚Δ• ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘• ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ
reverse proxyApp serverDBstructured log
fluentdͰૹ• ֤ϗετͰ in_tail +out_forward Ͱඈ͚ͩ͢• ͜͜΄ͱΜͲ͍͡Βͳ͍• தԝͷ aggregator ʹͯ͠Α͠ͳʹॲཧ• ࣭ϕετΤϑΥʔτ• ඞཁͳॴʹผ్อূͨ͠ΈΛཱͯΔ• ͍ͭͰʹϦόʔεϓϩΩγ͔ΒૹΓ࢝ΊΔ
reverse proxyApp serverDBaccess logerror logstructured logaggregator
֤ॴʹૹ• S3 ʹੜσʔλΛอଘ• Կ͔ίέͯ࠶ݱ• شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ
ϩάͷૹ• Mackerelʹߘ• ֤छϝτϦοΫՄࢹԽ• ϨΠςϯγ• ΤϥʔϨʔτ• KPI
ϩάͷૹ• Slackʹ௨• ΤϥʔϩάΛूΊΔ• ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……
ϩάͷૹ• Elasticsearch→Kibana ͰՄࢹԽ• ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ
ϩάͷૹ• MongoDB capped collection ʹೖ• ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ• ʢ࠷ۙ͋·ΓΘΕ͍ͯͳ͍ʣ
શܠ
ࠓޙ• syslog ͔Βͷ٫• ։ൃऀҎ֎׆༻Ͱ͖ΔΈ࡞Γ• ߦಈϩάج൫
ຊͷ༰• ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λհ͠·ͨ͠• ࠓޙಓͳվળΛଓ͚͍͖ͯ·͢
Ҏ্Ͱ͢• Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ http://hatenacorp.jp/recruit/career/operation-engineer• ౦ژɾژͲͪΒͰ
࣭ٙɾTwitterΑΓ• fluentd aggregator ٧·Γ·ͤΜ͔• aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝ͱʹ fluentd Λཱͯͯอଘ͍ͯ͠·͢• ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ͍ͯ͠·͢
࣭ٙɾTwitterΑΓ• ߦಈϩάͰܽଛ߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢• কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ֊Ͱ͢…• ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬͯܽଛ͠ͳ͍ΈΛ࡞Γ·͢ɻKafka ͷΑ͏ͳΩϡʔΛڬΉͳͲ
࣭ٙɾTwitterΑΓ• յΕͨSSDͲ͜ͷͰ͔͢• ॳ Crucial M4 Λ͍ͬͯ·ͨ͠