Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Triglav - Data Driven Workflow Tool
Search
Naotoshi Seo
February 17, 2017
1
4.1k
Triglav - Data Driven Workflow Tool
Naotoshi Seo
February 17, 2017
Tweet
Share
More Decks by Naotoshi Seo
See All by Naotoshi Seo
ZOZOTOWNリプレイス2020
sonots
5
35k
Red Chainer and Cumo: Practical Deep Learning in Ruby at RubyKaigi 2019
sonots
1
4.3k
Introduction of Cumo, and Integration to Red Chainer
sonots
1
1.1k
Implementation of Cumo, a CUDA-aware version of Ruby/Numo
sonots
1
1.9k
Fast Numerical Computing and Deep Learning in Ruby with Cumo
sonots
0
9.2k
CuPy improvments around memory
sonots
3
1.6k
DeNA AIシステム部におけるクラウドを活用した機械学習基盤の構築
sonots
4
6.1k
DeNA流データエンジニアリングの極意
sonots
17
13k
BigQuery Schema Migration #bq_sushi
sonots
2
6.1k
Featured
See All Featured
Raft: Consensus for Rubyists
vanstee
137
6.8k
How to train your dragon (web standard)
notwaldorf
90
5.8k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
666
120k
Six Lessons from altMBA
skipperchong
27
3.6k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
132
33k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
Designing for humans not robots
tammielis
250
25k
Code Review Best Practice
trishagee
66
17k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
31
2.1k
Site-Speed That Sticks
csswizardry
3
360
The Language of Interfaces
destraynor
156
24k
Become a Pro
speakerdeck
PRO
26
5.1k
Transcript
1 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBW %BUB%SJWFO8PSLqPX
https://github.com/triglav-dataflow
2 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 044 https://github.com/triglav-dataflow
˞υΩϡϝϯτগʑ͓ͪԼ͍͞
Importdb MySQL HDFS Vertica Medjed 01:00 06:00 Argus 10:00 3
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ❌Α͋͘ΔࠔΓ͝ͱ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ
Importdb MySQL HDFS Vertica Medjed 01:00 Argus Triglav Monitor Event
Monitor Event 4 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ⭕ରॲ๏ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ
5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Α͋͘ΔϫʔΫϑϩʔπʔϧͰμϝʁ •
શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ • ྫ શೖΓ+FOLJOTεϨʔϒ • ݹ ͍ʹ͑͠ ͷπʔϧͷଘࡏ • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍ • 31$ "1* Λ࣋ͨͳ͍ • +FOLJOTಉ࢜ͷ࿈ܞ ❌ࠔΓ͝ͱ %JHEBH -VJHJ +FOLJOTͳͲ৭ʑ͋Δ
5SJHMBWΞʔΩςΫνϟ ొਓ 6 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
DB agent consumer Triglav Core (Broker) 2. send message 1. polling message 1. watch every minute • .FTTBHF2VFVF • "HFOU 1SPEVDFS • ετϨʔδΛࢹ • $POTVNFS • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ • QVMMܕ • $PSF #SPLFS • ΠϕϯτΛཧ • QVMM͞Ε͙ͯ͢ʹΠϕϯτ ফ͑ͳ͍ σϑΥϧτอ࣋
HDFS Vertica BigQuery Triglav agent agent agent Medjed Jenkins Argus
Consumer Agent (Producer) Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBWΞʔΩςΫνϟ 7 • 5SJHMBWΛ)VCʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ • $POTVNFS5SJHMBWʹ͚ͩରԠ͢Εྑ͍
4XBHHFS 8 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
5SJHMBWͷ"1*༷Λ4XBHHFSͰهड़ • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB Λࣗಈੜ͍ͯ͠Δ RESTful APIͷ༷Λهड़͢Δ༷ͱͦͷπʔϧ܈
USJHMBWBHFOUIEGT KSVCZ 9 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• ࢹରͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ • ຖࢹ͢Δ • MTM3QBUIUP:NE • σϑΥϧτͰ • )BEPPQͳΒ*/PUJGZ"1*Λར༻͠ ͯΠϕϯτۦಈͰߋ৽ใΛऔΕΔ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕTFOENFTTBHF
USJHMBWBHFOUWFSUJDB DSVCZ 10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• جຊతʹUSJHMBWBHFOUIEGTͱಉ͡ • ςʔϒϧͷߦ͕͍࣋ͬͯΔFQPDIใΛར༻ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕύʔςΟγϣϯ୯ҐͰ·ͱΊ ͯTFOENFTTBHF
KFOLJOTUSJHHFSQPMMJOH@USJHMBW 11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
+FOLJOTϓϥάΠϯ • 5SJHMBWʹQPMMJOHͯ͠δϣϒ࣮ߦ
12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • .Z42-Ͱ.FTTBHF2VFVF࣮
ΧϥϜ໊ આ໌ ิ id ϝοηʔδID resource_uri σʔλϦιʔεͷURI ྫ) hdfs://path/to/%Y-%m-%d MQͷτϐ οΫʹ૬ resource_unit daily or hourly resource_timezone λΠϜκʔϯ ex) +09:00 resource_time Ϧιʔεߋ৽࣌ؒ 4&-&$5 GSPNNFTTBHFTXIFSFJE \PGGTFU^"/%SFTPVSDF@VSJ\VSJ^ 5SJHMBW࣮ৄࡉ
5SJHMBWύϑΥʔϚϯε 13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
USJHMBWBHFOU IEGTcWFSUJDB • ຖࢹ͢Δ͕ສఔͳ͠ • 4FOENFTTBHFසൟʹൃੜ͠ͳ͍ͨΊ ΄ͱΜͲؾʹͳΒͳ͍ • IPVSMZϦιʔεͰʹ௨ৗճ • ςʔϒϧ͋ͬͯ ఔ • 'FUDINFTTBHFຖߦΘΕΔ͕ɺ ΞϓϦαʔόNZTRMͷεϨʔϒΛ૿͢ ͳͲͰεέʔϧΞτ༰қ
'VUVSF8PSL 14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
ґଘͷՄࢹԽ • 'SPOU6*ͷ࡞ • ϩΰ • USJHMBWDPOTVNFSDMJ • Πϕϯτ͕དྷͨΒҾͷίϚϯυΛ࣮ߦ