Share
Embed
Transcript
25 slides
-
Agenda • ϝϧΧϦͷσʔλੳج൫ͷհ • Server log analysis infrastructure • Event
based log analysis infrastructure • Machine learning analysis infrastructure -
σʔλੳʹؔ͢Δׂ୲ @ • BI AnalystʢData Scientistʣ • ੳʹඞཁͳϩάσʔλߏͷઃܭ • σʔλੳʹج͍ͮͨاըఏҊɺKPIઃܭͱϨϙʔςΟϯάɾՄࢹԽ
• Software Engineer (Backend System, Site Reliability) • σʔλੳج൫ͷ։ൃɺӡ༻ɺαϙʔτ • Software Engineer (Machine Learning / NLP / AI) • (ओʹ)ػցֶशΛ͡Ίͱ͢Δઐٕज़ͱαʔϏεͷڮ͠Λߦ͏ • ݚڀ։ൃ݉ͶΔ -
Agenda • ϝϧΧϦͷσʔλੳج൫ • Server log analysis infrastructure • Event
based log analysis infrastructure • Machine learning analysis infrastructure -
Server log analysis infrastructure app app app access_log application_log app_error_log
error_log php_log... AWS S Check to make sure you recent set of AWS Simple This version was last upda (v1.4) Find the most recen aws.amazon.com/architect Usage Guidelines DEC 01 BigQuery AWS Check to make sure y recent set of AWS Sim This version was last u (v1.4) Find the most re aws.amazon.com/arch Always use Icon labe always include a label b the group in Arial. The Usage Guidelines DEC 01 Mackerel A Check to recent se This vers (v1.4) Fin aws.ama Always u always in the group Usage Guidel DEC 01 Slack Stream Processing batch Filtering & Import logs to BigQuery -
Server log analysis infrastructure • ֤αʔόͷϩάΛFluentdͰऩूɾసૹ • ༻్ʹԠ֤ͯ͡αʔϏεϛυϧΣΞʹೖ • BigQueryɿੳ༻ͷϩάશ෦͜͜ʹूΊΔ
• NorikraɿSQLʹΑΔετϦʔϛϯάॲཧ • etc…ʢe.g. KibanaɺKPI reportingʣ -
• όονͰϩάϑΝΠϧΛΠϯϙʔτ • over 1TB / day • ϩάϑΝΠϧࣗମGCSɺS3ʹόοΫΞοϓ •
Google Cloud SDK & AWS CLI & Embulk Google BigQuery -
• σʔλੳͷى • ։ൃऀ͕ௐࠪతͰΫΤϦΛ͛Δ • μογϡϘʔυ֤छεϓϨουγʔτͷσʔλιʔε • ͦͷଞϩάσʔλΛ׆༻ͨ͠෦͚αʔϏεͰར༻ • ఆֹϓϥϯΛར༻
• ΫΤϦͷྉۚΛؾʹ͠ͳͯ͘Α͍ • ͨͩ͠ɺεϩοτ༗ݶͳͷͰௐࢠʹͬͯॏ͍ΫΤϦΛ͛ա͗Δͱେ෯ʹԆ • தؒςʔϒϧΛ࡞͢Δ͜ͱͰॲཧྔΛݮ • εϩοτͷར༻ྔStackdriverͰ֬ೝͰ͖ΔʢᮢΞϥʔτՄʣ Google BigQuery -
• Google Spread Sheet • ͔ΒσʔλΛμϯϩʔυͯ͠ूܭɺάϥϑԽ • Excelܗࣜ׳Ε͍ͯΔਓ͕ଟ͍ͷͰɺඇΤϯδχΞͱͷڞ༗͕ḿΔ • Google
App Script • ొͨ͠ΫΤϦΛఆظతʹࣗಈ࣮ߦͯ͠ΦϨΦϨμογϡϘʔυੜ • ੜͨ͠άϥϑը૾Λ ʹߘ Google Spread Sheet & App Script -
• ͷΟϯυͰSQLʹΑΔूܭॲཧ • APIͷϦΫΤετ / minͷάϥϑԽ • 1ຖʹnginxͷΩϟογϡώοτΛMackerelʹ͛ͯάϥϑԽ • ̋ؒʹ˚݅Ҏ্ಛఆͷΤϥʔ͕ग़ͨΒSlackʹ௨
• etc… • Powered by fluentd-plugin-(mackerel|slack|norikra) Norikra -
Visualize and Alerting by log analysis Mackerel Slack Filter Aggregate
Summarize by SQL Visualize Alerting -
Stream processing by SQL 4&-&$5 $06/5 VQTUSFBN@DBDIF@TUBUVT)*5 $06/5
"4SBUF@IJU $06/5 VQTUSFBN@DBDIF@TUBUVT.*44 $06/5 "4SBUF@NJTT $06/5 VQTUSFBN@DBDIF@TUBUVT&91*3&% $06/5 "4SBUF@FYQJSFE '30.NFSDBSJ@TPNF@MPHXJOUJNF@CBUDI NJO 8)&3&VSJlTPNFBQJz ͱ͋ΔnginxͷΩϟογϡώοτ / min -
Agenda • ϝϧΧϦͷσʔλੳج൫ • Server log analysis infrastructure • Event
based log analysis infrastructure • Machine learning analysis infrastructure -
0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ Developer Data Sientist Analyze by SQL send
events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) fluent-agent-hydra Pascal - Event based log analysis infrastructure in_tail & out_forward BigQuery -
• Only Server log analysis infrastructure࣌ͷ՝(࣌) • ֤छKPIੳʹඞཁͳݩσʔλ͕ࢄ • ϩά͕ੜͷঢ়ଶʹ͍ۙͷͰ͍ͮΒ͍(ίπ͕͍Δ)
• ֎෦ͷੳπʔϧͩͱखܰʹͰ͖ΔҰํͰࡉ͔͘ௐͨΓɺෳ ͷσʔλπʔϧͱΈ߹ΘͤΔͷ͕͍͠ • ੳʹదͨ͠ϩάΛҰ͔Βઃܭɾूܭͯ͠ੳπʔϧͱΈ߹Θͤ ͯ͑ΔΑ͏ʹ͠Α͏ʂ • ৽͍͠ϩάੳج൫Λߏங͢Δ͜ͱʹ Pascal - Event based log analysis infrastructure -
0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ Developer Data Sientist Analyze by SQL send
events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) fluent-agent-hydra Pascal - Event based log analysis infrastructure in_tail & out_forward BigQuery -
• over 10,000 records / sec (not requests /sec) •
ΠϕϯτϕʔεͷϩάΛνϟϯωϧ୯ҐͰूɾసૹ • ΞϓϦ্ͷΠϕϯτϩάʢྫɿλοϓʣ • ։෧ϩά • ABςετϩά • etc… • ΞϓϦ͔Β͚ͩͰͳ֤͘छαϒγεςϜ͔Βͷϩάूɾసૹ Pascal - Event based log analysis infrastructure -
• over 10,000 records / sec (not requests /sec) •
ΠϕϯτϕʔεͷϩάΛνϟϯωϧ୯ҐͰूɾసૹ • ΞϓϦ্ͷΠϕϯτϩάʢྫɿλοϓʣ • ։෧ϩά • ABςετϩά • etc… • ΞϓϦ͔Β͚ͩͰͳ֤͘छαϒγεςϜ͔Βͷϩάूɾసૹ Pascal - Event based log analysis infrastructure -
A/B Testing new features • A/BςετͷϑϨʔϜϫʔΫԽʹΑΔॊೈੑͷ֬อ • े݅୯ҐͰA/BςετΛಉ࣌ਐߦ • A/BςετҎ֎ͷར༻ͷํ͋Δ
• ஈ֊తϦϦʔεʢ10% -> 50% -> 100%) • ػೳࣗମͷOn/Off • ݁ՌͷੳGoogle BigQuery -
Agenda • ϝϧΧϦͷσʔλੳج൫ • Server log analysis infrastructure • Event
based log analysis infrastructure • Machine learning analysis infrastructure -
• ٕज़ελοΫ • PythonɺDjangoɺscikit-learnɺTensorFlow • BigQuery্ͷσʔλΛݩʹ • σϞάϥϑΟοΫਪఆ • ΧςΰϦਪఆ
• ϥϕϦϯά • ৭ʑͳՕॴ͔Βར༻Ͱ͖ΔΑ͏ʹAPIͱͯ͠ఏڙ • ڈͷΕ͙Β͍ʹઐνʔϜ͕Ͱ͖ͯຊ֨తʹՔಇத Machine learning analysis infrastructure -
Summary • ϝϧΧϦͷσʔλੳج൫ • Server log analysis infrastructure • Event
based log analysis infrastructure • Machine learning analysis infrastructure • ϩάσʔλੳͷىGoogle BigQuery • ৭ʑͳπʔϧαʔϏεͱ࿈ܞ • A/BςετΈ͓ͯ͘͠ͱ৭ʑͱԠ༻͕ར͘ • ػցֶशΛར༻ͨ͠γεςϜຊ֨తʹՔಇ։࢝