Triglav - Data Driven Workflow Tool

Transcript

1. 1 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBW
   %BUB%SJWFO
   8PSLqPX

   https://github.com/triglav-dataflow

2. 2 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 044 https://github.com/triglav-dataflow

   ˞υΩϡϝϯτ͸গʑ͓଴ͪԼ͍͞

3. Importdb MySQL HDFS Vertica Medjed 01:00 06:00 Argus 10:00 3

   Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ͸ ❌Α͋͘ΔࠔΓ͝ͱ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ
   • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ

4. Importdb MySQL HDFS Vertica Medjed 01:00 Argus Triglav Monitor Event

   Monitor Event 4 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ͸ ⭕ରॲ๏ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ

5. 5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Α͋͘ΔϫʔΫϑϩʔπʔϧͰ͸μϝʁ •

   શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ
   • ྫ
   શೖΓ+FOLJOTεϨʔϒ
   • ݹ ͍ʹ͑͠ ͷπʔϧͷଘࡏ
   • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍
   • 31$ "1* Λ࣋ͨͳ͍
   • +FOLJOTಉ࢜ͷ࿈ܞ ❌ࠔΓ͝ͱ %JHEBH
   -VJHJ
   +FOLJOTͳͲ৭ʑ͋Δ

6. 5SJHMBWΞʔΩςΫνϟ ొ৔ਓ෺ 6 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.

   DB agent consumer Triglav Core (Broker) 2. send message 1. polling message 1. watch every minute • .FTTBHF
   2VFVF
   • "HFOU
   1SPEVDFS
   • ετϨʔδΛ؂ࢹ
   • $POTVNFS
   • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ
   • QVMM
   ܕ
   • $PSF
   #SPLFS
   • ΠϕϯτΛ؅ཧ
   • QVMM͞Εͯ΋͙͢ʹΠϕϯτ͸ ফ͑ͳ͍
   σϑΥϧτ೔อ࣋

7. HDFS Vertica BigQuery Triglav agent agent agent Medjed Jenkins Argus

   Consumer Agent (Producer) Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBWΞʔΩςΫνϟ 7 • 5SJHMBW
   Λ
   )VC
   ʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ • $POTVNFS͸5SJHMBW
   ʹ͚ͩରԠ͢Ε͹ྑ͍

8. 4XBHHFS 8 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •

   5SJHMBW
   ͷ
   "1*
   ࢓༷Λ
   4XBHHFS
   Ͱهड़
   • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB
   Λࣗಈੜ੒͍ͯ͠Δ RESTful APIͷ࢓༷Λهड़͢Δ࢓༷ͱͦͷπʔϧ܈

9. USJHMBWBHFOUIEGT
   KSVCZ 9 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.

   • ؂ࢹର৅ͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ
   • ຖ෼؂ࢹ͢Δ
   • MT
   M3
   QBUIUP:NE
   • σϑΥϧτͰ͸
   ೔෼
   • )BEPPQ
   
   
   ͳΒ
   */PUJGZ
   "1*
   Λར༻͠ ͯΠϕϯτۦಈͰߋ৽৘ใΛऔΕΔ
   • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕ͹
   TFOE
   NFTTBHF

10. USJHMBWBHFOUWFSUJDB
    DSVCZ 10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.

    • جຊతʹ͸
    USJHMBWBHFOUIEGT
    ͱಉ͡
    • ςʔϒϧͷߦ͕͍࣋ͬͯΔ
    FQPDI
    ৘ใΛར༻
    • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕ͹ύʔςΟγϣϯ୯ҐͰ·ͱΊ ͯ
    TFOE
    NFTTBHF

11. KFOLJOTUSJHHFSQPMMJOH@USJHMBW 11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •

    +FOLJOT
    ϓϥάΠϯ
    • 5SJHMBW
    ʹ
    QPMMJOH
    ͯ͠δϣϒ࣮ߦ

12. 12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • .Z42-Ͱ
    .FTTBHF
    2VFVF
    ࣮૷

    ΧϥϜ໊ આ໌ ิ଍ id ϝοηʔδID resource_uri σʔλϦιʔεͷURI ྫ) hdfs://path/to/%Y-%m-%d MQͷτϐ οΫʹ૬౰ resource_unit daily or hourly resource_timezone λΠϜκʔϯ ex) +09:00 resource_time Ϧιʔεߋ৽࣌ؒ 4&-&$5
    
    GSPN
    NFTTBHFT
    XIFSF
    JE
    
    \PGGTFU^
    "/%
    SFTPVSDF@VSJ
    
    \VSJ^ 5SJHMBW࣮૷ৄࡉ

13. 5SJHMBWύϑΥʔϚϯε 13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •

    USJHMBWBHFOU IEGTcWFSUJDB
    • ຖ෼؂ࢹ͢Δ͕ສఔ౓͸໰୊ͳ͠
    • 4FOE
    NFTTBHF
    ͸සൟʹൃੜ͠ͳ͍ͨΊ
    ΄ͱΜͲؾʹͳΒͳ͍
    • IPVSMZ
    ϦιʔεͰ΋೔ʹ௨ৗճ
    • ςʔϒϧ͋ͬͯ΋ ೔ఔ౓
    • 'FUDI
    NFTTBHF
    ͸ຖ෼ߦΘΕΔ͕ɺ
    ΞϓϦαʔό΍NZTRM
    ͷεϨʔϒΛ૿΍͢ ͳͲͰεέʔϧΞ΢τ༰қ

14. 'VUVSF
    8PSL 14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •

    ґଘͷՄࢹԽ
    • 'SPOU
    6*ͷ࡞੒
    • ϩΰ
    • USJHMBWDPOTVNFSDMJ
    • Πϕϯτ͕དྷͨΒҾ਺ͷίϚϯυΛ࣮ߦ