メルカリのデータ分析基盤 / mercari data analysis infrastructure

Transcript

1. Tatsuhiko Kubo@cubicdaiya σʔλ෼ੳج൫Night #2 2017/04/26 ͷσʔλ෼ੳج൫ / mercari log analysis
2. @cubicdaiya / Tatsuhiko Kubo Principal Engineer, SRE @ Mercari, Inc.
3. Agenda • ϝϧΧϦͷσʔλ෼ੳج൫ͷ঺հ • Server log analysis infrastructure • Event based log analysis infrastructure • Machine learning analysis infrastructure
4. σʔλ෼ੳʹؔ͢Δ໾ׂ෼୲ @ • BI AnalystʢData Scientistʣ • ෼ੳʹඞཁͳϩάσʔλߏ଄ͷઃܭ • σʔλ෼ੳʹج͍ͮͨاը΍ఏҊɺKPIઃܭͱϨϙʔςΟϯάɾՄࢹԽ • Software Engineer (Backend System, Site Reliability) • σʔλ෼ੳج൫ͷ։ൃɺӡ༻ɺαϙʔτ • Software Engineer (Machine Learning / NLP / AI) • (ओʹ)ػցֶशΛ͸͡Ίͱ͢Δઐ໳ٕज़ͱαʔϏεͷڮ౉͠Λߦ͏ • ݚڀ։ൃ΋݉ͶΔ
5. Agenda • ϝϧΧϦͷσʔλ෼ੳج൫ • Server log analysis infrastructure • Event based log analysis infrastructure • Machine learning analysis infrastructure
6. Server log analysis infrastructure app app app access_log application_log app_error_log error_log php_log... AWS S Check to make sure you recent set of AWS Simple This version was last upda (v1.4) Find the most recen aws.amazon.com/architect Usage Guidelines DEC 01 BigQuery AWS Check to make sure y recent set of AWS Sim This version was last u (v1.4) Find the most re aws.amazon.com/arch Always use Icon labe always include a label b the group in Arial. The Usage Guidelines DEC 01 Mackerel A Check to recent se This vers (v1.4) Fin aws.ama Always u always in the group Usage Guidel DEC 01 Slack Stream Processing batch Filtering & Import logs to BigQuery
7. Server log analysis infrastructure • ֤αʔόͷϩάΛFluentdͰऩूɾసૹ • ༻్ʹԠ֤ͯ͡αʔϏε΍ϛυϧ΢ΣΞʹ౤ೖ • BigQueryɿ෼ੳ༻ͷϩά͸શ෦͜͜ʹूΊΔ • NorikraɿSQLʹΑΔετϦʔϛϯάॲཧ • etc…ʢe.g. KibanaɺKPI reportingʣ
8. • όονͰϩάϑΝΠϧΛΠϯϙʔτ • over 1TB / day • ϩάϑΝΠϧࣗମ͸GCSɺS3ʹόοΫΞοϓ • Google Cloud SDK & AWS CLI & Embulk Google BigQuery
9. • σʔλ෼ੳͷى఺ • ։ൃऀ͕ௐࠪ໨తͰΫΤϦΛ౤͛Δ • μογϡϘʔυ΍֤छεϓϨουγʔτͷσʔλιʔε • ͦͷଞϩάσʔλΛ׆༻ͨ͠಺෦޲͚αʔϏεͰར༻ • ఆֹϓϥϯΛར༻ • ΫΤϦͷྉۚΛؾʹ͠ͳͯ͘Α͍ • ͨͩ͠ɺεϩοτ͸༗ݶͳͷͰௐࢠʹ৐ͬͯॏ͍ΫΤϦΛ౤͛ա͗Δͱେ෯ʹ஗Ԇ • தؒςʔϒϧΛ࡞੒͢Δ͜ͱͰॲཧྔΛ࡟ݮ • εϩοτͷར༻ྔ͸StackdriverͰ֬ೝͰ͖Δʢᮢ஋Ξϥʔτ΋Մʣ Google BigQuery
10. • ChartioΛར༻ • https://chartio.com/ • Ϋϥ΢υܕͷBIαʔϏε • ৭ʑͳσʔλιʔε͔Β μογϡϘʔυΛ࡞੒ Dashboard
11. • Google Spread Sheet • ͔ΒσʔλΛμ΢ϯϩʔυͯ͠ूܭɺάϥϑԽ • Excelܗࣜ͸׳Ε͍ͯΔਓ͕ଟ͍ͷͰɺඇΤϯδχΞͱͷڞ༗͕ḿΔ • Google App Script • ొ࿥ͨ͠ΫΤϦΛఆظతʹࣗಈ࣮ߦͯ͠ΦϨΦϨμογϡϘʔυੜ੒ • ੜ੒ͨ͠άϥϑը૾Λ ʹ౤ߘ Google Spread Sheet & App Script
12. • ਺෼ͷ΢Οϯυ΢ͰSQLʹΑΔूܭॲཧ • APIͷϦΫΤετ਺ / minͷάϥϑԽ • 1෼ຖʹnginxͷΩϟογϡώοτ཰ΛMackerelʹ౤͛ͯάϥϑԽ • ̋෼ؒʹ˚݅Ҏ্ಛఆͷΤϥʔ͕ग़ͨΒSlackʹ௨஌ • etc… • Powered by fluentd-plugin-(mackerel|slack|norikra) Norikra
13. Visualize and Alerting by log analysis Mackerel Slack Filter Aggregate Summarize by SQL Visualize Alerting
14. Stream processing by SQL 4&-&$5
    
    
    $06/5 
    VQTUSFBN@DBDIF@TUBUVT
    
    )*5
    $06/5  
    "4
    SBUF@IJU
    
    
    $06/5 
    VQTUSFBN@DBDIF@TUBUVT
    
    .*44 $06/5  
    "4
    SBUF@NJTT
    
    
    $06/5 
    VQTUSFBN@DBDIF@TUBUVT
    
    &91*3&% $06/5  
    "4
    SBUF@FYQJSFE
    '30.
    NFSDBSJ@TPNF@MPHXJOUJNF@CBUDI 
    NJO
    8)&3&
    VSJ
    
    lTPNFBQJz
    ͱ͋ΔnginxͷΩϟογϡώοτ཰ / min
15. Agenda • ϝϧΧϦͷσʔλ෼ੳج൫ • Server log analysis infrastructure • Event based log analysis infrastructure • Machine learning analysis infrastructure
16. 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ Developer Data Sientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) fluent-agent-hydra Pascal - Event based log analysis infrastructure in_tail & out_forward BigQuery
17. • Only Server log analysis infrastructure࣌୅ͷ՝୊(౰࣌) • ֤छKPI΍෼ੳʹඞཁͳݩσʔλ͕෼ࢄ • ϩά͕ੜͷঢ়ଶʹ͍ۙͷͰ࢖͍ͮΒ͍(ίπ͕͍Δ) • ֎෦ͷ෼ੳπʔϧͩͱखܰʹͰ͖ΔҰํͰࡉ͔͘ௐ΂ͨΓɺෳ ਺ͷσʔλ΍πʔϧͱ૊Έ߹ΘͤΔͷ͕೉͍͠ • ෼ੳʹదͨ͠ϩάΛҰ͔Βઃܭɾूܭͯ͠෼ੳπʔϧͱ૊Έ߹Θͤ ͯ࢖͑ΔΑ͏ʹ͠Α͏ʂ • ৽͍͠ϩά෼ੳج൫Λߏங͢Δ͜ͱʹ Pascal - Event based log analysis infrastructure
18. 0QFO3FTUZ 0QFO3FTUZ 0QFO3FTUZ Developer Data Sientist Analyze by SQL send events send events send events Powered by cookpad/puree-(ios|android) utilize events utilize events utilize events hydra(※) hydra(※) hydra(※) (※) fluent-agent-hydra Pascal - Event based log analysis infrastructure in_tail & out_forward BigQuery
19. • over 10,000 records / sec (not requests /sec) • ΠϕϯτϕʔεͷϩάΛνϟϯωϧ୯ҐͰू໿ɾసૹ • ΞϓϦ্ͷΠϕϯτϩάʢྫɿλοϓʣ • ։෧ϩά • ABςετϩά • etc… • ΞϓϦ͔Β͚ͩͰͳ֤͘छαϒγεςϜ͔Βͷϩά΋ू໿ɾసૹ Pascal - Event based log analysis infrastructure
20. • over 10,000 records / sec (not requests /sec) • ΠϕϯτϕʔεͷϩάΛνϟϯωϧ୯ҐͰू໿ɾసૹ • ΞϓϦ্ͷΠϕϯτϩάʢྫɿλοϓʣ • ։෧ϩά • ABςετϩά • etc… • ΞϓϦ͔Β͚ͩͰͳ֤͘छαϒγεςϜ͔Βͷϩά΋ू໿ɾసૹ Pascal - Event based log analysis infrastructure
21. A/B Testing new features • A/BςετͷϑϨʔϜϫʔΫԽʹΑΔॊೈੑͷ֬อ • ਺े݅୯ҐͰA/BςετΛಉ࣌ਐߦ • A/BςετҎ֎ͷར༻ͷ࢓ํ΋͋Δ • ஈ֊తϦϦʔεʢ10% -> 50% -> 100%) • ػೳࣗମͷOn/Off • ݁Ռͷ෼ੳ͸Google BigQuery
22. Agenda • ϝϧΧϦͷσʔλ෼ੳج൫ • Server log analysis infrastructure • Event based log analysis infrastructure • Machine learning analysis infrastructure
23. • ٕज़ελοΫ • PythonɺDjangoɺscikit-learnɺTensorFlow • BigQuery্ͷσʔλΛݩʹ • σϞάϥϑΟοΫਪఆ • ΧςΰϦਪఆ • ϥϕϦϯά • ৭ʑͳՕॴ͔Βར༻Ͱ͖ΔΑ͏ʹAPIͱͯ͠ఏڙ • ڈ೥ͷ฻Ε͙Β͍ʹઐ೚νʔϜ͕Ͱ͖ͯຊ֨తʹՔಇத Machine learning analysis infrastructure
24. Summary • ϝϧΧϦͷσʔλ෼ੳج൫ • Server log analysis infrastructure • Event based log analysis infrastructure • Machine learning analysis infrastructure • ϩάσʔλ෼ੳͷى఺͸Google BigQuery • ৭ʑͳπʔϧ΍αʔϏεͱ࿈ܞ • A/Bςετ͸࢓૊Έ͓ͯ͘͠ͱ৭ʑͱԠ༻͕ར͘ • ػցֶशΛར༻ͨ͠γεςϜ΋ຊ֨తʹՔಇ։࢝
25. We are hiring! https://www.mercari.com/jp/jobs/