Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Triglav - Data Driven Workflow Tool

Naotoshi Seo
February 17, 2017
3.7k

Triglav - Data Driven Workflow Tool

Naotoshi Seo

February 17, 2017
Tweet

Transcript

  1. 1
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    5SJHMBW
    %BUB%SJWFO8PSLqPX

    https://github.com/triglav-dataflow

    View Slide

  2. 2
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    044
    https://github.com/triglav-dataflow
    ˞υΩϡϝϯτ͸গʑ͓଴ͪԼ͍͞

    View Slide

  3. Importdb
    MySQL HDFS Vertica
    Medjed
    01:00 06:00
    Argus
    10:00
    3
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    τϦάϥϑͱ͸
    ❌Α͋͘ΔࠔΓ͝ͱ
    • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋
    ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ
    • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ
    • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ

    View Slide

  4. Importdb
    MySQL HDFS Vertica
    Medjed
    01:00
    Argus
    Triglav
    Monitor Event
    Monitor Event
    4
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    τϦάϥϑͱ͸
    ⭕ରॲ๏
    • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋
    ͨͤͯϫʔΫϑϩʔܗ੒Λࢧԉ͢Δπʔϧ

    View Slide

  5. 5
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    Α͋͘ΔϫʔΫϑϩʔπʔϧͰ͸μϝʁ
    • શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ
    • ྫ
    શೖΓ+FOLJOTεϨʔϒ
    • ݹ ͍ʹ͑͠
    ͷπʔϧͷଘࡏ
    • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍

    • 31$ "1*
    Λ࣋ͨͳ͍
    • +FOLJOTಉ࢜ͷ࿈ܞ
    ❌ࠔΓ͝ͱ
    %JHEBH -VJHJ +FOLJOTͳͲ৭ʑ͋Δ

    View Slide

  6. 5SJHMBWΞʔΩςΫνϟ ొ৔ਓ෺

    6
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    DB
    agent consumer
    Triglav Core (Broker)
    2. send message
    1. polling
    message
    1. watch
    every minute
    • .FTTBHF2VFVF
    • "HFOU 1SPEVDFS

    • ετϨʔδΛ؂ࢹ
    • $POTVNFS
    • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ
    • QVMMܕ
    • $PSF #SPLFS

    • ΠϕϯτΛ؅ཧ
    • QVMM͞Εͯ΋͙͢ʹΠϕϯτ͸
    ফ͑ͳ͍ σϑΥϧτ೔อ࣋

    View Slide

  7. HDFS
    Vertica
    BigQuery
    Triglav
    agent
    agent
    agent
    Medjed
    Jenkins
    Argus
    Consumer
    Agent (Producer)
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    5SJHMBWΞʔΩςΫνϟ
    7
    • 5SJHMBWΛ)VCʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ
    • $POTVNFS͸5SJHMBWʹ͚ͩରԠ͢Ε͹ྑ͍

    View Slide

  8. 4XBHHFS
    8
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • 5SJHMBWͷ"1*࢓༷Λ4XBHHFSͰهड़
    • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB
    Λࣗಈੜ੒͍ͯ͠Δ
    RESTful APIͷ࢓༷Λهड़͢Δ࢓༷ͱͦͷπʔϧ܈

    View Slide

  9. USJHMBWBHFOUIEGT KSVCZ

    9
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ؂ࢹର৅ͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ
    • ຖ෼؂ࢹ͢Δ
    • MTM3QBUIUP:NE
    • σϑΥϧτͰ͸೔෼
    • )BEPPQͳΒ*/PUJGZ"1*Λར༻͠
    ͯΠϕϯτۦಈͰߋ৽৘ใΛऔΕΔ
    • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ
    λ͕ߋ৽͞Ε͍ͯΕ͹TFOENFTTBHF

    View Slide

  10. USJHMBWBHFOUWFSUJDB DSVCZ

    10
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • جຊతʹ͸USJHMBWBHFOUIEGTͱಉ͡
    • ςʔϒϧͷߦ͕͍࣋ͬͯΔFQPDI৘ใΛར༻
    • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ
    λ͕ߋ৽͞Ε͍ͯΕ͹ύʔςΟγϣϯ୯ҐͰ·ͱΊ
    ͯTFOENFTTBHF

    View Slide

  11. [email protected]
    11
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • +FOLJOTϓϥάΠϯ
    • 5SJHMBWʹQPMMJOHͯ͠δϣϒ࣮ߦ

    View Slide

  12. 12
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • .Z42-Ͱ.FTTBHF2VFVF࣮૷
    ΧϥϜ໊ આ໌ ิ଍
    id ϝοηʔδID
    resource_uri σʔλϦιʔεͷURI
    ྫ) hdfs://path/to/%Y-%m-%d
    MQͷτϐ
    οΫʹ૬౰
    resource_unit daily or hourly
    resource_timezone λΠϜκʔϯ ex) +09:00
    resource_time Ϧιʔεߋ৽࣌ؒ
    4&-&$5GSPNNFTTBHFTXIFSFJE
    \PGGTFU^"/%[email protected]\VSJ^
    5SJHMBW࣮૷ৄࡉ

    View Slide

  13. 5SJHMBWύϑΥʔϚϯε
    13
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • USJHMBWBHFOU IEGTcWFSUJDB

    • ຖ෼؂ࢹ͢Δ͕ສఔ౓͸໰୊ͳ͠
    • 4FOENFTTBHF͸සൟʹൃੜ͠ͳ͍ͨΊ
    ΄ͱΜͲؾʹͳΒͳ͍
    • IPVSMZϦιʔεͰ΋೔ʹ௨ৗճ
    • ςʔϒϧ͋ͬͯ΋ ೔ఔ౓
    • 'FUDINFTTBHF͸ຖ෼ߦΘΕΔ͕ɺ
    ΞϓϦαʔό΍NZTRMͷεϨʔϒΛ૿΍͢
    ͳͲͰεέʔϧΞ΢τ༰қ

    View Slide

  14. 'VUVSF8PSL
    14
    Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
    • ґଘͷՄࢹԽ
    • 'SPOU6*ͷ࡞੒
    • ϩΰ
    • USJHMBWDPOTVNFSDMJ
    • Πϕϯτ͕དྷͨΒҾ਺ͷίϚϯυΛ࣮ߦ

    View Slide