Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Triglav - Data Driven Workflow Tool
Search
Naotoshi Seo
February 17, 2017
1
3.9k
Triglav - Data Driven Workflow Tool
Naotoshi Seo
February 17, 2017
Tweet
Share
More Decks by Naotoshi Seo
See All by Naotoshi Seo
ZOZOTOWNリプレイス2020
sonots
5
31k
Red Chainer and Cumo: Practical Deep Learning in Ruby at RubyKaigi 2019
sonots
1
3.8k
Introduction of Cumo, and Integration to Red Chainer
sonots
1
930
Implementation of Cumo, a CUDA-aware version of Ruby/Numo
sonots
1
1.6k
Fast Numerical Computing and Deep Learning in Ruby with Cumo
sonots
0
8.5k
CuPy improvments around memory
sonots
3
1.4k
DeNA AIシステム部におけるクラウドを活用した機械学習基盤の構築
sonots
4
5.8k
DeNA流データエンジニアリングの極意
sonots
17
12k
BigQuery Schema Migration #bq_sushi
sonots
2
5.9k
Featured
See All Featured
Web development in the modern age
philhawksworth
201
10k
Become a Pro
speakerdeck
PRO
8
4.4k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
225
51k
RailsConf 2023
tenderlove
0
510
5 minutes of I Can Smell Your CMS
philhawksworth
199
19k
A designer walks into a library…
pauljervisheath
199
23k
Making Projects Easy
brettharned
106
5.4k
Six Lessons from altMBA
skipperchong
19
2.9k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
28
5.9k
Building Your Own Lightsaber
phodgson
97
5.6k
Principles of Awesome APIs and How to Build Them.
keavy
119
16k
For a Future-Friendly Web
brad_frost
170
8.9k
Transcript
1 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBW %BUB%SJWFO8PSLqPX
https://github.com/triglav-dataflow
2 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 044 https://github.com/triglav-dataflow
˞υΩϡϝϯτগʑ͓ͪԼ͍͞
Importdb MySQL HDFS Vertica Medjed 01:00 06:00 Argus 10:00 3
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ❌Α͋͘ΔࠔΓ͝ͱ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ
Importdb MySQL HDFS Vertica Medjed 01:00 Argus Triglav Monitor Event
Monitor Event 4 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ⭕ରॲ๏ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ
5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Α͋͘ΔϫʔΫϑϩʔπʔϧͰμϝʁ •
શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ • ྫ શೖΓ+FOLJOTεϨʔϒ • ݹ ͍ʹ͑͠ ͷπʔϧͷଘࡏ • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍ • 31$ "1* Λ࣋ͨͳ͍ • +FOLJOTಉ࢜ͷ࿈ܞ ❌ࠔΓ͝ͱ %JHEBH -VJHJ +FOLJOTͳͲ৭ʑ͋Δ
5SJHMBWΞʔΩςΫνϟ ొਓ 6 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
DB agent consumer Triglav Core (Broker) 2. send message 1. polling message 1. watch every minute • .FTTBHF2VFVF • "HFOU 1SPEVDFS • ετϨʔδΛࢹ • $POTVNFS • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ • QVMMܕ • $PSF #SPLFS • ΠϕϯτΛཧ • QVMM͞Ε͙ͯ͢ʹΠϕϯτ ফ͑ͳ͍ σϑΥϧτอ࣋
HDFS Vertica BigQuery Triglav agent agent agent Medjed Jenkins Argus
Consumer Agent (Producer) Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBWΞʔΩςΫνϟ 7 • 5SJHMBWΛ)VCʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ • $POTVNFS5SJHMBWʹ͚ͩରԠ͢Εྑ͍
4XBHHFS 8 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
5SJHMBWͷ"1*༷Λ4XBHHFSͰهड़ • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB Λࣗಈੜ͍ͯ͠Δ RESTful APIͷ༷Λهड़͢Δ༷ͱͦͷπʔϧ܈
USJHMBWBHFOUIEGT KSVCZ 9 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• ࢹରͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ • ຖࢹ͢Δ • MTM3QBUIUP:NE • σϑΥϧτͰ • )BEPPQͳΒ*/PUJGZ"1*Λར༻͠ ͯΠϕϯτۦಈͰߋ৽ใΛऔΕΔ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕTFOENFTTBHF
USJHMBWBHFOUWFSUJDB DSVCZ 10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• جຊతʹUSJHMBWBHFOUIEGTͱಉ͡ • ςʔϒϧͷߦ͕͍࣋ͬͯΔFQPDIใΛར༻ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕύʔςΟγϣϯ୯ҐͰ·ͱΊ ͯTFOENFTTBHF
KFOLJOTUSJHHFSQPMMJOH@USJHMBW 11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
+FOLJOTϓϥάΠϯ • 5SJHMBWʹQPMMJOHͯ͠δϣϒ࣮ߦ
12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • .Z42-Ͱ.FTTBHF2VFVF࣮
ΧϥϜ໊ આ໌ ิ id ϝοηʔδID resource_uri σʔλϦιʔεͷURI ྫ) hdfs://path/to/%Y-%m-%d MQͷτϐ οΫʹ૬ resource_unit daily or hourly resource_timezone λΠϜκʔϯ ex) +09:00 resource_time Ϧιʔεߋ৽࣌ؒ 4&-&$5 GSPNNFTTBHFTXIFSFJE \PGGTFU^"/%SFTPVSDF@VSJ\VSJ^ 5SJHMBW࣮ৄࡉ
5SJHMBWύϑΥʔϚϯε 13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
USJHMBWBHFOU IEGTcWFSUJDB • ຖࢹ͢Δ͕ສఔͳ͠ • 4FOENFTTBHFසൟʹൃੜ͠ͳ͍ͨΊ ΄ͱΜͲؾʹͳΒͳ͍ • IPVSMZϦιʔεͰʹ௨ৗճ • ςʔϒϧ͋ͬͯ ఔ • 'FUDINFTTBHFຖߦΘΕΔ͕ɺ ΞϓϦαʔόNZTRMͷεϨʔϒΛ૿͢ ͳͲͰεέʔϧΞτ༰қ
'VUVSF8PSL 14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
ґଘͷՄࢹԽ • 'SPOU6*ͷ࡞ • ϩΰ • USJHMBWDPOTVNFSDMJ • Πϕϯτ͕དྷͨΒҾͷίϚϯυΛ࣮ߦ