Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Triglav - Data Driven Workflow Tool
Search
Naotoshi Seo
February 17, 2017
1
4.1k
Triglav - Data Driven Workflow Tool
Naotoshi Seo
February 17, 2017
Tweet
Share
More Decks by Naotoshi Seo
See All by Naotoshi Seo
ZOZOTOWNリプレイス2020
sonots
5
35k
Red Chainer and Cumo: Practical Deep Learning in Ruby at RubyKaigi 2019
sonots
1
4.3k
Introduction of Cumo, and Integration to Red Chainer
sonots
1
1.1k
Implementation of Cumo, a CUDA-aware version of Ruby/Numo
sonots
1
1.9k
Fast Numerical Computing and Deep Learning in Ruby with Cumo
sonots
0
9.2k
CuPy improvments around memory
sonots
3
1.6k
DeNA AIシステム部におけるクラウドを活用した機械学習基盤の構築
sonots
4
6.1k
DeNA流データエンジニアリングの極意
sonots
17
13k
BigQuery Schema Migration #bq_sushi
sonots
2
6.1k
Featured
See All Featured
The World Runs on Bad Software
bkeepers
PRO
67
11k
Intergalactic Javascript Robots from Outer Space
tanoku
270
27k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
Making the Leap to Tech Lead
cromwellryan
133
9.1k
VelocityConf: Rendering Performance Case Studies
addyosmani
328
24k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
33
2.8k
A better future with KSS
kneath
238
17k
Git: the NoSQL Database
bkeepers
PRO
427
65k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.2k
We Have a Design System, Now What?
morganepeng
51
7.4k
Building a Modern Day E-commerce SEO Strategy
aleyda
38
7.1k
Building Adaptive Systems
keathley
40
2.4k
Transcript
1 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBW %BUB%SJWFO8PSLqPX
https://github.com/triglav-dataflow
2 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 044 https://github.com/triglav-dataflow
˞υΩϡϝϯτগʑ͓ͪԼ͍͞
Importdb MySQL HDFS Vertica Medjed 01:00 06:00 Argus 10:00 3
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ❌Α͋͘ΔࠔΓ͝ͱ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ
Importdb MySQL HDFS Vertica Medjed 01:00 Argus Triglav Monitor Event
Monitor Event 4 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ⭕ରॲ๏ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ
5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Α͋͘ΔϫʔΫϑϩʔπʔϧͰμϝʁ •
શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ • ྫ શೖΓ+FOLJOTεϨʔϒ • ݹ ͍ʹ͑͠ ͷπʔϧͷଘࡏ • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍ • 31$ "1* Λ࣋ͨͳ͍ • +FOLJOTಉ࢜ͷ࿈ܞ ❌ࠔΓ͝ͱ %JHEBH -VJHJ +FOLJOTͳͲ৭ʑ͋Δ
5SJHMBWΞʔΩςΫνϟ ొਓ 6 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
DB agent consumer Triglav Core (Broker) 2. send message 1. polling message 1. watch every minute • .FTTBHF2VFVF • "HFOU 1SPEVDFS • ετϨʔδΛࢹ • $POTVNFS • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ • QVMMܕ • $PSF #SPLFS • ΠϕϯτΛཧ • QVMM͞Ε͙ͯ͢ʹΠϕϯτ ফ͑ͳ͍ σϑΥϧτอ࣋
HDFS Vertica BigQuery Triglav agent agent agent Medjed Jenkins Argus
Consumer Agent (Producer) Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBWΞʔΩςΫνϟ 7 • 5SJHMBWΛ)VCʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ • $POTVNFS5SJHMBWʹ͚ͩରԠ͢Εྑ͍
4XBHHFS 8 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
5SJHMBWͷ"1*༷Λ4XBHHFSͰهड़ • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB Λࣗಈੜ͍ͯ͠Δ RESTful APIͷ༷Λهड़͢Δ༷ͱͦͷπʔϧ܈
USJHMBWBHFOUIEGT KSVCZ 9 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• ࢹରͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ • ຖࢹ͢Δ • MTM3QBUIUP:NE • σϑΥϧτͰ • )BEPPQͳΒ*/PUJGZ"1*Λར༻͠ ͯΠϕϯτۦಈͰߋ৽ใΛऔΕΔ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕTFOENFTTBHF
USJHMBWBHFOUWFSUJDB DSVCZ 10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• جຊతʹUSJHMBWBHFOUIEGTͱಉ͡ • ςʔϒϧͷߦ͕͍࣋ͬͯΔFQPDIใΛར༻ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕύʔςΟγϣϯ୯ҐͰ·ͱΊ ͯTFOENFTTBHF
KFOLJOTUSJHHFSQPMMJOH@USJHMBW 11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
+FOLJOTϓϥάΠϯ • 5SJHMBWʹQPMMJOHͯ͠δϣϒ࣮ߦ
12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • .Z42-Ͱ.FTTBHF2VFVF࣮
ΧϥϜ໊ આ໌ ิ id ϝοηʔδID resource_uri σʔλϦιʔεͷURI ྫ) hdfs://path/to/%Y-%m-%d MQͷτϐ οΫʹ૬ resource_unit daily or hourly resource_timezone λΠϜκʔϯ ex) +09:00 resource_time Ϧιʔεߋ৽࣌ؒ 4&-&$5 GSPNNFTTBHFTXIFSFJE \PGGTFU^"/%SFTPVSDF@VSJ\VSJ^ 5SJHMBW࣮ৄࡉ
5SJHMBWύϑΥʔϚϯε 13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
USJHMBWBHFOU IEGTcWFSUJDB • ຖࢹ͢Δ͕ສఔͳ͠ • 4FOENFTTBHFසൟʹൃੜ͠ͳ͍ͨΊ ΄ͱΜͲؾʹͳΒͳ͍ • IPVSMZϦιʔεͰʹ௨ৗճ • ςʔϒϧ͋ͬͯ ఔ • 'FUDINFTTBHFຖߦΘΕΔ͕ɺ ΞϓϦαʔόNZTRMͷεϨʔϒΛ૿͢ ͳͲͰεέʔϧΞτ༰қ
'VUVSF8PSL 14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
ґଘͷՄࢹԽ • 'SPOU6*ͷ࡞ • ϩΰ • USJHMBWDPOTVNFSDMJ • Πϕϯτ͕དྷͨΒҾͷίϚϯυΛ࣮ߦ