サービスに寄り添うログ基盤/pepabo_log_infrastructure_bigfoot

サービスに寄り添うログ基盤 - ログ収集のその先に -
はてな・ペパボ技術大会〜インフラ技術基盤〜@京都
http://hatena.connpass.com/event/33521/

Transcript

1. ϩάऩूͷͦͷઌʹ
   ࡾ୐༔հ(.01&1"#0JOD
   ͸ͯͳɾϖύϘٕज़େձʙΠϯϑϥٕज़ج൫ʙ!ژ౎
   αʔϏεʹدΓఴ͏ϩάج൫
   

   View Slide

2. ϓϦϯγύϧΤϯδχΞ
   ࡾ୐༔հ!NPOPDISPNFHBOF
   NJOOFࣄۀ෦
   IUUQCMPHNPOPDISPNFHBOFDPN
   

   View Slide

3. NJOOF
   IUUQTNJOOFDPN
   

   View Slide

4. ໨࣍
   w8FCαʔϏεͱߦಈϩά
   w#JHGPPU
   wαʔϏεʹدΓఴ͏ϩάج൫
   

   View Slide

5. 8FCαʔϏεͱߦಈϩά
   

   View Slide

6. ϩά͸͍͍ͧ
   

   View Slide

7. ߦಈϩά
   

   View Slide

8. ߦಈϩά
   ΞϓϦέʔγϣϯ૚Ͱग़ྗ͢Δϩά
   ͍ͭɺͩΕ͕ɺͳʹΛ΍͔͕ͬͨಛఆͰ͖Δ
   ࠷ऴతͳߦಈ݁Ռ͚ͩͰͳ͘ɺ్தͷͲ͜Ͱ͖͋ΒΊ͔ͨɺͲ͏໎͔ͬͨ
   ͕Θ͔Δ
   

   View Slide

9. ߦಈϩάʹ͸
   αʔϏεվળͷώϯτ͕ͭ·͍ͬͯΔ
   

   View Slide

10. ߦಈϩάͷ׆༻ஈ֊
    

    View Slide

11. ߦಈϩάͷ׆༻ஈ֊
    ऩूߦಈϩά͕ग़ྗ͞ΕɺऔΓ·ͱΊΒΕ͍ͯΔঢ়ଶ
    ෼ੳऔΓ·ͱΊͨߦಈϩάΛࢹ֮Խɺ෼ੳͰ͖Δঢ়ଶ
    ׆༻෼ੳͨ͠ߦಈϩάΛ΋ͱʹܧଓతͳαʔϏεվળ͕ߦ͍͑ͯΔঢ়ଶ
    

    View Slide

12. ϩάج൫
    

    View Slide

13. ϩάج൫ʹେ੾ͳ͜ͱ
    

    View Slide

14. lϩάͷ׆༻z
    

    View Slide

15. ϩάl׆༻zج൫
    

    View Slide

16. #JHGPPU
    

    View Slide

17. #JHGPPU
    wϖύϘͷ࣍ੈ୅ϩάl׆༻zج൫
    wߦಈϩάͷऩूɺ෼ੳɺ׆༻ͷ֤ஈ֊ʹ͓͍ͯɺશࣾͰར༻Ͱ͖Δ൚༻ੑͱ۩
    ମతͳ׆༻ํ๏Λఏڙ
    wࠃ಺࠷େڃϋϯυϝΠυϚʔέοτNJOOFΛࢧ͑Δϩάج൫
    

    View Slide

18. #JHGPPU
    IDFA/GAID
    UID
    rack-bigfoot
    Service
    Request
    Activity
    log
    Services
    DB
    Attribute
    Big Cube
    Cube
    https://icons8.com
    BI
    Recommendation
    Bandit algorithm
    Re-marketing
    Feedback
    Name identification
    Cookie Sync
    

    View Slide

19. #JHGPPUΛࢧ͑Δٕज़
    

    View Slide

20. ऩूɾ෼ੳ
    

    View Slide

21. ϩάΛૹΔ
    

    View Slide

22. SBDLCJHGPPU
    w3BJMTΞϓϦέʔγϣϯͱ'MVFOUEΛͭͳ͙3BDLϛυϧ΢ΣΞ
    w#JHGPPUʹඞཁͳڞ௨ύϥϝλΛϦΫΤετɾϨεϙϯεϔομ͔Βऔಘ
    wαʔϏεݻ༗ͷύϥϝλΛ෇༩͢Δ͜ͱ΋Մೳ
    Rails.application.config.app_middleware.insert_after ActionDispatch::Callbacks,
    Rack::Bigfoot do |config|
    config.service = 'minne'
    config.environment = Rails.env
    config.enable_fluent = Rails.env.production? || Rails.env.staging?
    config.ignore_path_patterns << %r(\A/healthcheck)
    config.headers << 'HTTP_X_CLIENT_VERSION'
    end
    

    View Slide

23. ϩάΛͨΊΔ
    

    View Slide

24. 5SFBTVSF%BUB
    wΫϥ΢υܕσʔλϚωδϝϯταʔϏε
    wIUUQTXXXUSFBTVSFEBUBDPN
    wେ༰ྔͷϩάอଘɺ෼ࢄॲཧʹΑΔߴ଎ͳϩάૢ࡞
    log Plasma DB
    HiveQL
    export
    SQL
    aggregate
    Data Tanks
    

    View Slide

25. ϩάΛѻ͏
    

    View Slide

26. )JWF2-
    w5SFBTVSF%BUB্ͷߦಈϩάΛ42-ϥΠΫʹѻ͏
    wIUUQIJWFBQBDIFPSH
    wIUUQTEPDTUSFBTVSFEBUBDPNBSUJDMFTIJWF
    SELECT
    TD_TIME_FORMAT(time, 'yyyy-MM-dd HH:mm:ss', 'JST') AS timestamp,
    response_time,
    request_method,
    path_info
    FROM
    activity
    WHERE
    TD_TIME_RANGE(time, '2016-07-01 10:00:00', '2016-07-01 12:00:00', 'JST');
    

    View Slide

27. ϫʔΫϑϩʔ
    w5SFBTVSF%BUBͷεέδϡʔϧΫΤϦΛར༻
    wΫΤϦͷίʔυ؅ཧ༻ʹ1FOEVMVNΛ։ൃ
    wIUUQTHJUIVCDPNNPOPDISPNFHBOFQFOEVMVN
    w%4-ʹΑͬͯεέδϡʔϧΫΤϦΛهड़͠ɺίʔυ؅ཧ
    Scheduled queries
    Queries on GitHub
    Apply
    Pendulum
    

    View Slide

28. 1FOEVMVN
    schedule 'test-scheduled-job' do
    database 'db_name'
    query 'select time from access;'
    retry_limit 0
    priority :normal
    cron '30 0 * * *'
    timezone 'Asia/Tokyo'
    delay 0
    result_url 'td://@/db_name/table_name'
    end
    Schedfile
    Apply
    $ pendulum --apikey='...' -a --dry-run
    $ pendulum --apikey='...' -a
    

    View Slide

29. %JHEBHҠߦத
    IUUQTHJUIVCDPNUSFBTVSFEBUBEJHEBH
    
    

    View Slide

30. ϩάΛศརʹ͢Δ
    

    View Slide

31. ଐੑ৘ใ
    wߦಈϩάͱଐੑ৘ใΛ૊Έ߹ΘͤΔ͜ͱͰ෼ੳ࣌ͷ෯͕޿͕Δ
    Attribute
    Master 1,000 records each
    Sidekiq workers
    def perform(*args)
    User.order(:id).select(:id).find_in_batches do |users|
    UserAttributesUploadJob.perform_later(users.first.id, users.last.id)
    end
    end
    Activity
    Join
    HiveQL
    No temporary file
    

    View Slide

32. ໊دͤ
    wαʔϏεͷΞΧ΢ϯτͱ֤ΫϥΠΞϯτΛϚοϐϯά
    wະϩάΠϯঢ়ଶͷΞΧ΢ϯτ΋໊دͤޙʹաڈʹḪͬͯඥ෇͚
    w$PPLJF4ZODͱ૊Έ߹ΘͤͯαʔϏεΛ·͍ͨͩϚοϐϯά΋Մೳ
    Name identification
    

    View Slide

33. ϩάΛ෼ੳ͢Δ
    

    View Slide

34. #JH$VCFͱ$VCF
    wશαʔϏεͷߦಈϩάΛू໿ͨ͠#JH$VCF
    w੾ޱ͕֬ఆͨ͠΋ͷ͸ϝδϟʔΧϥϜɺσΟ
    ϝϯγϣϯΧϥϜͷ୯ҐͰ$VCFʹ੾Γग़͠
    wϝδϟʔఆྔԽՄೳͳΧϥϜ
    wσΟϝϯγϣϯूܭͷ੾ΓޱͱͳΔΧϥϜ
    wྫ࣌ؒ͝ͱͷച্ɺ౎ಓ෎ݝ͝ͱͷ࡞඼਺
    w$VCF͸σʔλϚʔτʹஔ͖ɺߴ଎ʹࢀরͰ
    ͖ΔΑ͏ʹ͢Δ
    Activity Big Cube
    Cube
    HiveQL SQL
    BI, Dashboard
    ad-hoc query
    Analyst Managers,
    Product owners,
    Promotion groups
    

    View Slide

35. ࢹ֮Խͱ෼ੳ
    wࢹ֮Խͱ෼ੳʹ͸5BCMFBVࣾͷ5BCMFBV%FTLUPQΛར༻
    wIUUQXXXUBCMFBVDPN
    wσʔλιʔεͱͯ͠5SFBTVSF%BUBΛબ୒Մ
    w μογϡϘʔυྫ
    w ྲྀ௨ֹɺΩϟϯηϧֹۚɺ஫จֹۚɺϢʔβʔ୯Ձ
    w ྦྷੵձһ਺ɺ஫จ୯Ձɺ৽نొ࿥Ϣʔβʔɺ%"6$৽نɺ%"6$طଘ
    w ड஫࡞඼਺ɺड஫཰ɺड஫࡞඼Ձ֨ɺड஫Մೳ࡞඼਺
    w ૯ࡏݿ਺ɺࡏݿ୯Ձɺࡏݿ૯ֹ
    w ड஫Մೳ࡞Ո਺ɺൢചத࡞඼਺ɺ։ళத࡞඼਺ɺ૯࡞඼਺
    

    View Slide

36. ׆༻
    

    View Slide

37. ׆༻
    w෼ੳͨ݁͠ՌΛ΋ͱʹԾઆΛཱͯͯγεςϜͷվमΛߦ͏
    wը໘σβΠϯͷมߋɺεςοϓͷݟ௚͠
    w"#ςετ
    ˠ੩తͳϑΟʔυόοΫ
    

    View Slide

38. ಈతͳϑΟʔυόοΫ
    

    View Slide

39. όϯσΟοτΞϧΰϦζϜ
    w୳ٻͱ׆༻ͷׂ߹Λߋ৽͠ଓ͚Δ͜ͱͰ"#ςετͷػձଛࣦΛݮΒ͢
    wIUUQTXXXPSFJMMZDPKQCPPLT
    wྫ͑͹ɺ͋Δػೳͷ$53Λվળ͢ΔͨΊʹׂ͸࠷ળͷख๏ʢ׆༻ʣɺ࢒Γ
    ׂͰෳ਺ͷख๏Λࢼ͢ʢ୳ٻʣ
    Activity
    Epsilon-Greedy algorithm
    User
    1-ε: exploitation
    ε/pattern: exploration
    Click or not click
    Import
    

    View Slide

40. Ϩίϝϯυ
    wNJOOFʮ͋ͳͨʹ͓͢͢Ίͷ࡞Ոʯ
    wϢʔβʔͷߦಈΛجʹ࡞ՈΛϨʔςΟϯά
    Activity
    Filter and shuffle
    Users
    fav, follow etc…
    Matrix Factorization
    Recommendation
    import
    DB
    

    View Slide

41. Ϩίϝϯυ.BUSJY'BDUPSJ[BUJPO
    wڠௐϑΟϧλϦϯάϢʔβͷᅂ޷৘ใΛ஝ੵ͠ɺ͋ΔϢʔβͱᅂ޷ͷྨࣅ͠
    ͨଞͷϢʔβͷ৘ใΛ༻͍ͯਪ࿦Λߦ͏
    w.BUSJY'BDUPSJ[BUJPO
    w࣍ݩ࡟ݮ
    wϢʔβʔ΍࡞඼͝ͱͷධՁͷภΓ͕͋Γɺૄͳσʔλʹର͢ΔධՁ༧ଌ
    
    
    
    
    
    Item
    User R ≈
    =
    m
    P
    n n Q
    ×
    m
    k
    k
    

    View Slide

42. Ϩίϝϯυ.BUSJY'BDUPSJ[BUJPO
    R’ui =
    μ + Bu
    + Bi
    + Pu
    TQi
    minP,Q,B
    Σ (Rui
    - R’ui
    )2 + λ(||Bu
    ||2 + ||Bi
    ||2 + ||Pu
    ||2 + ||Qi
    ||2)
    ༧ଌ
    ֶश
    (u,i)∈R
    ਖ਼ଇԽ߲
    ޡࠩ
    ฏۉ όΠΞε
    

    View Slide

43. )JWFNBMM
    

    View Slide

44. Ϩίϝϯυ.BUSJY'BDUPSJ[BUJPO
    SELECT
    idx,
    array_avg(u_rank) as Pu,
    array_avg(i_rank) as Qi,
    avg(u_bias) as Bu,
    avg(i_bias) as Bi,
    min(mu) as mu
    FROM (
    SELECT train_mf_sgd(account_id, creator_id, rating,
    '-factor 20 -iter 50 -update_mu') AS (idx, u_rank,
    i_rank, u_bias, i_bias, mu)
    FROM training
    ) t
    GROUP BY idx;
    

    View Slide

45. ͳͲͳͲ
    w͍ΘΏΔɺӾཡ์غɺΧʔτ์غͷ࡞඼Λߦಈϩά͔Βநग़
    wಛఆͷ৚݅Ͱݺͼ໭͠ͷ௨஌Λߦ͏
    wߦಈϩά͔Βؔ࿈ੑͷߴ͍޿ࠂΛग़͢
    wϦϚʔέςΟϯά
    w޿ࠂର৅ͷηάϝϯτԽʢߜࠐɺআ֎ʣ
    CSPXTF DBSUBCBOEPONFOU
    ޿ࠂ࿈ܞ
    

    View Slide

46. αʔϏεʹدΓఴ͏ϩάج൫
    

    View Slide

47. αʔϏεʹدΓఴ͏ϩάج൫
    w୯ʹϩάΛूΊΔ͚ͩʹͤͣɺ෼ੳɺ׆༻ͷஈ֊Λิॿ͢Δ
    w੩తͳϑΟʔυόοΫ͔ΒಈతͳϑΟʔυόοΫ΁
    wߦಈϩάͷ॥؀ʹΑΓɺͳΊΒ͔ͳੈք΁
    

    View Slide

48. ϩά͸͍͍ͧ
    

    View Slide

49. ͓ΘΓ
    

    View Slide

50. ܅΋ϖύϘͰಇ͔ͳ͍͔ʁ
    ࠷৽ͷ࠾༻৘ใΛνΣοΫˠ !QC@SFDSVJU
    

    View Slide