Triglav - Data Driven Workflow Tool

Aef92c3acc29ad8543e04135687fc4f1?s=47 Naotoshi Seo
February 17, 2017
3.1k

Triglav - Data Driven Workflow Tool

Aef92c3acc29ad8543e04135687fc4f1?s=128

Naotoshi Seo

February 17, 2017
Tweet

Transcript

  1. 1 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBW %BUB%SJWFO8PSLqPX

    https://github.com/triglav-dataflow
  2. 2 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 044 https://github.com/triglav-dataflow

    ˞υΩϡϝϯτ͸গʑ͓଴ͪԼ͍͞
  3. Importdb MySQL HDFS Vertica Medjed 01:00 06:00 Argus 10:00 3

    Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ͸ ❌Α͋͘ΔࠔΓ͝ͱ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ
  4. Importdb MySQL HDFS Vertica Medjed 01:00 Argus Triglav Monitor Event

    Monitor Event 4 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ͸ ⭕ରॲ๏ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ
  5. 5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Α͋͘ΔϫʔΫϑϩʔπʔϧͰ͸μϝʁ •

    શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ • ྫ શೖΓ+FOLJOTεϨʔϒ • ݹ ͍ʹ͑͠ ͷπʔϧͷଘࡏ • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍  • 31$ "1* Λ࣋ͨͳ͍ • +FOLJOTಉ࢜ͷ࿈ܞ ❌ࠔΓ͝ͱ %JHEBH -VJHJ +FOLJOTͳͲ৭ʑ͋Δ
  6. 5SJHMBWΞʔΩςΫνϟ ొ৔ਓ෺ 6 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.

    DB agent consumer Triglav Core (Broker) 2. send message 1. polling message 1. watch every minute • .FTTBHF2VFVF • "HFOU 1SPEVDFS  • ετϨʔδΛ؂ࢹ • $POTVNFS • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ • QVMMܕ • $PSF #SPLFS  • ΠϕϯτΛ؅ཧ • QVMM͞Εͯ΋͙͢ʹΠϕϯτ͸ ফ͑ͳ͍ σϑΥϧτ೔อ࣋
  7. HDFS Vertica BigQuery Triglav agent agent agent Medjed Jenkins Argus

    Consumer Agent (Producer) Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBWΞʔΩςΫνϟ 7 • 5SJHMBWΛ)VCʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ • $POTVNFS͸5SJHMBWʹ͚ͩରԠ͢Ε͹ྑ͍
  8. 4XBHHFS 8 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •

    5SJHMBWͷ"1*࢓༷Λ4XBHHFSͰهड़ • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB Λࣗಈੜ੒͍ͯ͠Δ RESTful APIͷ࢓༷Λهड़͢Δ࢓༷ͱͦͷπʔϧ܈
  9. USJHMBWBHFOUIEGT KSVCZ 9 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.

    • ؂ࢹର৅ͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ • ຖ෼؂ࢹ͢Δ • MTM3QBUIUP:NE • σϑΥϧτͰ͸೔෼ • )BEPPQͳΒ*/PUJGZ"1*Λར༻͠ ͯΠϕϯτۦಈͰߋ৽৘ใΛऔΕΔ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕ͹TFOENFTTBHF
  10. USJHMBWBHFOUWFSUJDB DSVCZ 10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.

    • جຊతʹ͸USJHMBWBHFOUIEGTͱಉ͡ • ςʔϒϧͷߦ͕͍࣋ͬͯΔFQPDI৘ใΛར༻ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕ͹ύʔςΟγϣϯ୯ҐͰ·ͱΊ ͯTFOENFTTBHF
  11. KFOLJOTUSJHHFSQPMMJOH@USJHMBW 11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •

    +FOLJOTϓϥάΠϯ • 5SJHMBWʹQPMMJOHͯ͠δϣϒ࣮ߦ
  12. 12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • .Z42-Ͱ.FTTBHF2VFVF࣮૷

    ΧϥϜ໊ આ໌ ิ଍ id ϝοηʔδID resource_uri σʔλϦιʔεͷURI ྫ) hdfs://path/to/%Y-%m-%d MQͷτϐ οΫʹ૬౰ resource_unit daily or hourly resource_timezone λΠϜκʔϯ ex) +09:00 resource_time Ϧιʔεߋ৽࣌ؒ 4&-&$5 GSPNNFTTBHFTXIFSFJE \PGGTFU^"/%SFTPVSDF@VSJ\VSJ^ 5SJHMBW࣮૷ৄࡉ
  13. 5SJHMBWύϑΥʔϚϯε 13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •

    USJHMBWBHFOU IEGTcWFSUJDB  • ຖ෼؂ࢹ͢Δ͕ສఔ౓͸໰୊ͳ͠ • 4FOENFTTBHF͸සൟʹൃੜ͠ͳ͍ͨΊ ΄ͱΜͲؾʹͳΒͳ͍ • IPVSMZϦιʔεͰ΋೔ʹ௨ৗճ • ςʔϒϧ͋ͬͯ΋ ೔ఔ౓ • 'FUDINFTTBHF͸ຖ෼ߦΘΕΔ͕ɺ ΞϓϦαʔό΍NZTRMͷεϨʔϒΛ૿΍͢ ͳͲͰεέʔϧΞ΢τ༰қ
  14. 'VUVSF8PSL 14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •

    ґଘͷՄࢹԽ • 'SPOU6*ͷ࡞੒ • ϩΰ • USJHMBWDPOTVNFSDMJ • Πϕϯτ͕དྷͨΒҾ਺ͷίϚϯυΛ࣮ߦ