Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Speaker Deck
PRO
Sign in
Sign up for free
はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6
tatsuru
PRO
August 31, 2016
7
11k
はてなのログ運用 これまでとこれから / Hatena Engineer Seminar #6
Hatena Engineer Seminar #6 インフラ編での発表スライドです
tatsuru
PRO
August 31, 2016
Tweet
Share
More Decks by tatsuru
See All by tatsuru
Mackerelのプロダクト開発 - エンジニア中心の開発プロセスで大切にしていること
tatsuru
PRO
0
2.9k
Mackerel の EventBridge 対応開発秘話
tatsuru
PRO
1
96
技術が実現するイノベーションとWebサービス運用の未来 / Innovation from&for Web Operations
tatsuru
PRO
0
1k
成長するためのエンジニア組織 / Hatena Engineering Group 2018
tatsuru
PRO
1
68
Mesosを使ったImmutable Infra 管理システムを作ってみた
tatsuru
PRO
8
7.2k
Featured
See All Featured
Debugging Ruby Performance
tmm1
67
11k
Practical Orchestrator
shlominoach
178
8.9k
10 Git Anti Patterns You Should be Aware of
lemiorhan
643
54k
Done Done
chrislema
178
14k
Designing on Purpose - Digital PM Summit 2013
jponch
108
5.9k
Design and Strategy: How to Deal with People Who Don’t "Get" Design
morganepeng
109
16k
Fantastic passwords and where to find them - at NoRuKo
philnash
32
1.9k
The Language of Interfaces
destraynor
149
21k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
182
15k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
318
19k
How GitHub Uses GitHub to Build GitHub
holman
465
280k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
13
1.1k
Transcript
ͯͳͷϩάӡ༻ ͜Ε·Ͱͱ͜Ε͔Β id:wtatsuru 2016/08/31 Hatena Engineer Seminar #6
ࣗݾհ • Tatsuru Watabane (id:wtatsuru) • ͯͳγεςϜϓϥοτϑΥʔϜ෦ • 2011ೖࣾ •
2014·ͰͷαʔϏε͍͍ͩͨશ෦ӡ༻
ຊͷ༰ • લఏͱͳΔγεςϜߏ • ͯͳͷϩάͷߏͱӡ༻Λհ • ࠓޙ
લఏͱͳΔγεςϜߏ
3ߏ reverse proxy App server DB
ѻ͏ϩά reverse proxy App server DB access log error log
various logs
ѻ͏ϩά • αʔϏεͷΞΫηεϩά • ˏϦόʔεϓϩΩγ • ΞϓϦέʔγϣϯ͕ు͘ϩά • Τϥʔϩάɺߦಈϩά etc.
reverse proxy App server DB access log error log various logs
લఏɿଟ͘ͷαʔϏε • ৽αʔϏε͔Β10Ҏ্ଓ͘αʔϏε·Ͱଟ༷ • ֤αʔϏεΛ·͍ͨͩڞ௨ج൫͕ඞཁ • ։ൃνʔϜͱҰॹʹվળͰ͖Δମ੍Λ࡞Γ͍ͨ
ͯͳͷϩάӡ༻
reverse proxy App server DB access log error log structured
log syslog server storage server SSD server rsyslog aggregator
ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ • ΞϓϦέʔγϣϯϩάͷӡ༻ • ߏஙͷྺ࢙ͱͱʹΏΔ͘հ͠·͢
ΞΫηεϩάग़ྗ • ΞΫηεϩάͷϚελͱͳΔϩάͷग़ྗ • ֤छௐࠪɺૢ࡞ϩάɺetc. • ϩʔΧϧʹϩάΛు͘@ϦόʔεϓϩΩγ • LTSV (Labeled
Tab-separated Values) http://ltsv.org/ host:127.0.0.1 ident:- user:frank time:[10/Oct/2000:13:55:36 -0700] req:GET /apache_pb.gif HTTP/1.0 status:200 size:2326 referer:http:// www.example.com/start.html ua:Mozilla/4.08 [en] (Win98; I ;Nav)
reverse proxy App server DB access log ΞΫηεϩάग़ྗ
ϩάΛอଘ͍ͨ͠ • ผͷαʔόʹอଘ͓͖ͯͨ͘͠ͳͬͨ • rsyslog Ͱඈͯ͠ετϨʔδαʔόʹอଘ • ϩʔΧϧ͔ΒS3ʹอଘͯ͠ೋॏԽ • ͠Β͘͢ΔͱGlacierʹΉ
reverse proxy App server DB access log syslog server rsyslog
storage server
؆୯ʹੳ͍ͨ͠ • SSDͷαʔόʹඈ͢ • SSD͕҆͘ͳͬͨͷͰಋೖ • ࡶଟʹ grep ੳ •
ΞΫηεϩά͔ΒϨΠςϯγੳ • 2ॻ͍ͨΒSSD͕յΕͨͷͰަ
reverse proxy App server DB access log syslog server storage
server SSD server rsyslog
ͬͱ؆୯ʹੳ͍ͨ͠ • S3 ͔Β Amazon EMR (ElasticMapReduce) • LTSVΛಡΈࠐΉσγϦΞϥΠβ
github.com/hatena/KeyValuePairsDeserializer • SQLͰ؆୯ʹੳ select count(distinct regexp_extract(referer,"http://([^/]+)/",1)) from logs where req rlike 'GET /hoge‘
reverse proxy App server DB access log syslog server storage
server SSD server rsyslog
ͬͱ؆୯ʹੳ͍ͨ͠ • ֤छ՝ • ॳظԽ͕͍ • ڥߏங͕͍͠ • ఆظతʹճ͢ͱίετ͕ •
ΘΕͳ͘ͳ͍ͬͯ͘…
ͬͱ؆୯ʹੳ͍ͨ͠ • EMRஔ͖͑ɿGoogle BigQuery Λݕূ • LTSV embulk Ͱલॲཧͯ͠ಡΈࠐΉ
• ͍ɾ·͋·͍͋҆ • ։ൃऀҎ֎Ͱ͍͍͢
reverse proxy App server DB access log syslog server storage
server SSD server rsyslog
ͯͳͷϩάӡ༻ • ΞΫηεϩάͷӡ༻ɹɹɹɹɹɹˡ͜͜·Ͱ • ΞϓϦέʔγϣϯϩάͷӡ༻ɹɹˡ͔͜͜Β • ߏஙͷྺ࢙ͱͱʹΏΔ͘հ͠·͢
ΞϓϦέʔγϣϯͷϩά • ૉͳੲ • /var/log/httpd/access.log • /var/log/httpd/error.log • ֤छͪ͝Ό·ͥ •
Τϥʔௐ͕ࠪͭΒ͍ɺूܭ͕ͭΒ͍ • DBʹೖΕΔ
reverse proxy App server DB
ߏԽϩάΛग़͢ • શαʔϏεͰಛఆͷ path ʹ JSON Ͱు͘ • ඪ४ग़ྗɾΤϥʔग़ྗͱ͚Δ •
ΞϓϦέʔγϣϯ͔Β໌ࣔతʹు͘ • ߦಈϩάɺDBΞΫηεɺͦͷଞͳΜͰ
reverse proxy App server DB structured log
fluentdͰૹ • ֤ϗετͰ in_tail +out_forward Ͱඈ͚ͩ͢ • ͜͜΄ͱΜͲ͍͡Βͳ͍ • தԝͷ
aggregator ʹͯ͠Α͠ͳʹॲཧ • ࣭ϕετΤϑΥʔτ • ඞཁͳॴʹผ్อূͨ͠ΈΛཱͯΔ • ͍ͭͰʹϦόʔεϓϩΩγ͔ΒૹΓ࢝ΊΔ
reverse proxy App server DB access log error log structured
log aggregator
֤ॴʹૹ • S3 ʹੜσʔλΛอଘ • Կ͔ίέͯ࠶ݱ • شൃੑͷอଘઌ͕ଟ͍ͨΊ͜͜Ͱ୲อ͢Δ
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • Mackerelʹߘ • ֤छϝτϦοΫՄࢹԽ • ϨΠςϯγ • ΤϥʔϨʔτ •
KPI
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • Slackʹ௨ • ΤϥʔϩάΛूΊΔ • ݹ͍αʔϏεͰϩά͕ଟ͍ˠ grep Ͱ៉ྷʹ……
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • Elasticsearch→Kibana ͰՄࢹԽ • ֤νʔϜͰμογϡϘʔυΛ࡞ͬͯݟΔ
reverse proxy App server DB access log error log structured
log aggregator
ϩάͷૹ • MongoDB capped collection ʹೖ • ͦͷ··ݟͨΓɺࡶʹूܭͨ͠Γ • ʢ࠷ۙ͋·ΓΘΕ͍ͯͳ͍ʣ
reverse proxy App server DB access log error log structured
log aggregator
શܠ
reverse proxy App server DB access log error log structured
log syslog server storage server SSD server rsyslog aggregator
ࠓޙ • syslog ͔Βͷ٫ • ։ൃऀҎ֎׆༻Ͱ͖ΔΈ࡞Γ • ߦಈϩάج൫
ຊͷ༰ • ͯͳͰͷϩάͷӡ༻ͷݱঢ়Λհ͠·ͨ͠ • ࠓޙಓͳվળΛଓ͚͍͖ͯ·͢
Ҏ্Ͱ͢ • Ұॹʹվળͯ͘͠ΕΔํΛืूதͰ͢ʂ http://hatenacorp.jp/recruit/career/operation-engineer • ౦ژɾژͲͪΒͰ
࣭ٙɾTwitterΑΓ • fluentd aggregator ٧·Γ·ͤΜ͔ • aggregator ͷઌʹɺͦΕͧΕͷσʔλετΞ͝ ͱʹ fluentd
Λཱͯͯอଘ͍ͯ͠·͢ • ͜͏͢Δ͜ͱͰӨڹΛσʔλετΞ୯ҐʹہॴԽ ͍ͯ͠·͢
࣭ٙɾTwitterΑΓ • ߦಈϩάͰܽଛ߹ੑͳͲ͍Ζ͍Ζߟ͑Δͱ͜ Ζ͋Δͱࢥ͍·͕͢Կ͔ߟ͍͑ͯ·͔͢ • কདྷతʹͦ͏͍͏ͷ͕Ͱ͖ͨΒ͍͍ͳɺͱ͍͏ஈ ֊Ͱ͢… • ࣮ࡍඞཁͱͳͬͨΒɺผʹ౸ୡอূΛͨ͠ܥΛ࡞ͬ ͯܽଛ͠ͳ͍ΈΛ࡞Γ·͢ɻKafka
ͷΑ͏ͳ ΩϡʔΛڬΉͳͲ
࣭ٙɾTwitterΑΓ • յΕͨSSDͲ͜ͷͰ͔͢ • ॳ Crucial M4 Λ͍ͬͯ·ͨ͠