Triglav - Data Driven Workflow Tool

Transcript

1. 1
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   5SJHMBW
   %BUB%SJWFO8PSLqPX
   
   https://github.com/triglav-dataflow
   

   View Slide

2. 2
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   044
   https://github.com/triglav-dataflow
   ˞υΩϡϝϯτ͸গʑ͓଴ͪԼ͍͞
   

   View Slide

3. Importdb
   MySQL HDFS Vertica
   Medjed
   01:00 06:00
   Argus
   10:00
   3
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   τϦάϥϑͱ͸
   ❌Α͋͘ΔࠔΓ͝ͱ
   • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋
   ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ
   • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ
   • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ
   
   

   View Slide

4. Importdb
   MySQL HDFS Vertica
   Medjed
   01:00
   Argus
   Triglav
   Monitor Event
   Monitor Event
   4
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   τϦάϥϑͱ͸
   ⭕ରॲ๏
   • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋
   ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ
   

   View Slide

5. 5
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   Α͋͘ΔϫʔΫϑϩʔπʔϧͰ͸μϝʁ
   • શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ
   • ྫ
   શೖΓ+FOLJOTεϨʔϒ
   • ݹ ͍ʹ͑͠
   ͷπʔϧͷଘࡏ
   • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍
   
   • 31$ "1*
   Λ࣋ͨͳ͍
   • +FOLJOTಉ࢜ͷ࿈ܞ
   ❌ࠔΓ͝ͱ
   %JHEBH -VJHJ +FOLJOTͳͲ৭ʑ͋Δ
   

   View Slide

6. 5SJHMBWΞʔΩςΫνϟ ొ৔ਓ෺
   
   6
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   DB
   agent consumer
   Triglav Core (Broker)
   2. send message
   1. polling
   message
   1. watch
   every minute
   • .FTTBHF2VFVF
   • "HFOU 1SPEVDFS
   
   • ετϨʔδΛ؂ࢹ
   • $POTVNFS
   • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ
   • QVMMܕ
   • $PSF #SPLFS
   
   • ΠϕϯτΛ؅ཧ
   • QVMM͞Εͯ΋͙͢ʹΠϕϯτ͸
   ফ͑ͳ͍ σϑΥϧτ೔อ࣋
   
   

   View Slide

7. HDFS
   Vertica
   BigQuery
   Triglav
   agent
   agent
   agent
   Medjed
   Jenkins
   Argus
   Consumer
   Agent (Producer)
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   5SJHMBWΞʔΩςΫνϟ
   7
   • 5SJHMBWΛ)VCʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ
   • $POTVNFS͸5SJHMBWʹ͚ͩରԠ͢Ε͹ྑ͍
   

   View Slide

8. 4XBHHFS
   8
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   • 5SJHMBWͷ"1*࢓༷Λ4XBHHFSͰهड़
   • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB
   Λࣗಈੜ੒͍ͯ͠Δ
   RESTful APIͷ࢓༷Λهड़͢Δ࢓༷ͱͦͷπʔϧ܈
   

   View Slide

9. USJHMBWBHFOUIEGT KSVCZ
   
   9
   Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
   • ؂ࢹର৅ͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ
   • ຖ෼؂ࢹ͢Δ
   • MTM3QBUIUP:NE
   • σϑΥϧτͰ͸೔෼
   • )BEPPQͳΒ*/PUJGZ"1*Λར༻͠
   ͯΠϕϯτۦಈͰߋ৽৘ใΛऔΕΔ
   • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ
   λ͕ߋ৽͞Ε͍ͯΕ͹TFOENFTTBHF
   

   View Slide

10. USJHMBWBHFOUWFSUJDB DSVCZ
    
    10
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • جຊతʹ͸USJHMBWBHFOUIEGTͱಉ͡
    • ςʔϒϧͷߦ͕͍࣋ͬͯΔFQPDI৘ใΛར༻
    • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ
    λ͕ߋ৽͞Ε͍ͯΕ͹ύʔςΟγϣϯ୯ҐͰ·ͱΊ
    ͯTFOENFTTBHF
    

    View Slide

11. KFOLJOTUSJHHFSQPMMJOH@USJHMBW
    11
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • +FOLJOTϓϥάΠϯ
    • 5SJHMBWʹQPMMJOHͯ͠δϣϒ࣮ߦ
    

    View Slide

12. 12
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • .Z42-Ͱ.FTTBHF2VFVF࣮૷
    ΧϥϜ໊ આ໌ ิ଍
    id ϝοηʔδID
    resource_uri σʔλϦιʔεͷURI
    ྫ) hdfs://path/to/%Y-%m-%d
    MQͷτϐ
    οΫʹ૬౰
    resource_unit daily or hourly
    resource_timezone λΠϜκʔϯ ex) +09:00
    resource_time Ϧιʔεߋ৽࣌ؒ
    4&-&$5GSPNNFTTBHFTXIFSFJE
    \PGGTFU^"/%SFTPVSDF@VSJ\VSJ^
    5SJHMBW࣮૷ৄࡉ
    

    View Slide

13. 5SJHMBWύϑΥʔϚϯε
    13
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • USJHMBWBHFOU IEGTcWFSUJDB
    
    • ຖ෼؂ࢹ͢Δ͕ສఔ౓͸໰୊ͳ͠
    • 4FOENFTTBHF͸සൟʹൃੜ͠ͳ͍ͨΊ
    ΄ͱΜͲؾʹͳΒͳ͍
    • IPVSMZϦιʔεͰ΋೔ʹ௨ৗճ
    • ςʔϒϧ͋ͬͯ΋ ೔ఔ౓
    • 'FUDINFTTBHF͸ຖ෼ߦΘΕΔ͕ɺ
    ΞϓϦαʔό΍NZTRMͷεϨʔϒΛ૿΍͢
    ͳͲͰεέʔϧΞ΢τ༰қ
    

    View Slide

14. 'VUVSF8PSL
    14
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ґଘͷՄࢹԽ
    • 'SPOU6*ͷ࡞੒
    • ϩΰ
    • USJHMBWDPOTVNFSDMJ
    • Πϕϯτ͕དྷͨΒҾ਺ͷίϚϯυΛ࣮ߦ
    

    View Slide