Triglav - Data Driven Workflow Tool

Transcript

1. 1 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBW
   %BUB%SJWFO
   8PSLqPX https://github.com/triglav-dataflow
2. 2 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 044 https://github.com/triglav-dataflow ˞υΩϡϝϯτ͸গʑ͓଴ͪԼ͍͞
3. Importdb MySQL HDFS Vertica Medjed 01:00 06:00 Argus 10:00 3 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ͸ ❌Α͋͘ΔࠔΓ͝ͱ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ
   • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ
4. Importdb MySQL HDFS Vertica Medjed 01:00 Argus Triglav Monitor Event Monitor Event 4 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ͸ ⭕ରॲ๏ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ
5. 5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Α͋͘ΔϫʔΫϑϩʔπʔϧͰ͸μϝʁ • શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ
   • ྫ
   શೖΓ+FOLJOTεϨʔϒ
   • ݹ ͍ʹ͑͠ ͷπʔϧͷଘࡏ
   • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍
   • 31$ "1* Λ࣋ͨͳ͍
   • +FOLJOTಉ࢜ͷ࿈ܞ ❌ࠔΓ͝ͱ %JHEBH
   -VJHJ
   +FOLJOTͳͲ৭ʑ͋Δ
6. 5SJHMBWΞʔΩςΫνϟ ొ৔ਓ෺ 6 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. DB agent consumer Triglav Core (Broker) 2. send message 1. polling message 1. watch every minute • .FTTBHF
   2VFVF
   • "HFOU
   1SPEVDFS
   • ετϨʔδΛ؂ࢹ
   • $POTVNFS
   • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ
   • QVMM
   ܕ
   • $PSF
   #SPLFS
   • ΠϕϯτΛ؅ཧ
   • QVMM͞Εͯ΋͙͢ʹΠϕϯτ͸ ফ͑ͳ͍
   σϑΥϧτ೔อ࣋
7. HDFS Vertica BigQuery Triglav agent agent agent Medjed Jenkins Argus Consumer Agent (Producer) Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBWΞʔΩςΫνϟ 7 • 5SJHMBW
   Λ
   )VC
   ʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ • $POTVNFS͸5SJHMBW
   ʹ͚ͩରԠ͢Ε͹ྑ͍
8. 4XBHHFS 8 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • 5SJHMBW
   ͷ
   "1*
   ࢓༷Λ
   4XBHHFS
   Ͱهड़
   • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB
   Λࣗಈੜ੒͍ͯ͠Δ RESTful APIͷ࢓༷Λهड़͢Δ࢓༷ͱͦͷπʔϧ܈
9. USJHMBWBHFOUIEGT
   KSVCZ 9 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ؂ࢹର৅ͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ
   • ຖ෼؂ࢹ͢Δ
   • MT
   M3
   QBUIUP:NE
   • σϑΥϧτͰ͸
   ೔෼
   • )BEPPQ
   
   
   ͳΒ
   */PUJGZ
   "1*
   Λར༻͠ ͯΠϕϯτۦಈͰߋ৽৘ใΛऔΕΔ
   • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕ͹
   TFOE
   NFTTBHF
10. USJHMBWBHFOUWFSUJDB
    DSVCZ 10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • جຊతʹ͸
    USJHMBWBHFOUIEGT
    ͱಉ͡
    • ςʔϒϧͷߦ͕͍࣋ͬͯΔ
    FQPDI
    ৘ใΛར༻
    • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕ͹ύʔςΟγϣϯ୯ҐͰ·ͱΊ ͯ
    TFOE
    NFTTBHF
11. KFOLJOTUSJHHFSQPMMJOH@USJHMBW 11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • +FOLJOT
    ϓϥάΠϯ
    • 5SJHMBW
    ʹ
    QPMMJOH
    ͯ͠δϣϒ࣮ߦ
12. 12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • .Z42-Ͱ
    .FTTBHF
    2VFVF
    ࣮૷ ΧϥϜ໊ આ໌ ิ଍ id ϝοηʔδID resource_uri σʔλϦιʔεͷURI ྫ) hdfs://path/to/%Y-%m-%d MQͷτϐ οΫʹ૬౰ resource_unit daily or hourly resource_timezone λΠϜκʔϯ ex) +09:00 resource_time Ϧιʔεߋ৽࣌ؒ 4&-&$5
    
    GSPN
    NFTTBHFT
    XIFSF
    JE
    
    \PGGTFU^
    "/%
    SFTPVSDF@VSJ
    
    \VSJ^ 5SJHMBW࣮૷ৄࡉ
13. 5SJHMBWύϑΥʔϚϯε 13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • USJHMBWBHFOU IEGTcWFSUJDB
    • ຖ෼؂ࢹ͢Δ͕ສఔ౓͸໰୊ͳ͠
    • 4FOE
    NFTTBHF
    ͸සൟʹൃੜ͠ͳ͍ͨΊ
    ΄ͱΜͲؾʹͳΒͳ͍
    • IPVSMZ
    ϦιʔεͰ΋೔ʹ௨ৗճ
    • ςʔϒϧ͋ͬͯ΋ ೔ఔ౓
    • 'FUDI
    NFTTBHF
    ͸ຖ෼ߦΘΕΔ͕ɺ
    ΞϓϦαʔό΍NZTRM
    ͷεϨʔϒΛ૿΍͢ ͳͲͰεέʔϧΞ΢τ༰қ
14. 'VUVSF
    8PSL 14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • ґଘͷՄࢹԽ
    • 'SPOU
    6*ͷ࡞੒
    • ϩΰ
    • USJHMBWDPOTVNFSDMJ
    • Πϕϯτ͕དྷͨΒҾ਺ͷίϚϯυΛ࣮ߦ