Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Triglav - Data Driven Workflow Tool
Search
Naotoshi Seo
February 17, 2017
1
4.1k
Triglav - Data Driven Workflow Tool
Naotoshi Seo
February 17, 2017
Tweet
Share
More Decks by Naotoshi Seo
See All by Naotoshi Seo
ZOZOTOWNリプレイス2020
sonots
5
35k
Red Chainer and Cumo: Practical Deep Learning in Ruby at RubyKaigi 2019
sonots
1
4.3k
Introduction of Cumo, and Integration to Red Chainer
sonots
1
1.1k
Implementation of Cumo, a CUDA-aware version of Ruby/Numo
sonots
1
1.9k
Fast Numerical Computing and Deep Learning in Ruby with Cumo
sonots
0
9.2k
CuPy improvments around memory
sonots
3
1.6k
DeNA AIシステム部におけるクラウドを活用した機械学習基盤の構築
sonots
4
6.1k
DeNA流データエンジニアリングの極意
sonots
17
13k
BigQuery Schema Migration #bq_sushi
sonots
2
6.1k
Featured
See All Featured
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
33
2.1k
What's in a price? How to price your products and services
michaelherold
244
12k
Visualization
eitanlees
146
15k
Reflections from 52 weeks, 52 projects
jeffersonlam
348
20k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
133
33k
Designing on Purpose - Digital PM Summit 2013
jponch
117
7.1k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
7
640
BBQ
matthewcrist
87
9.5k
Speed Design
sergeychernyshev
27
810
Docker and Python
trallard
44
3.3k
YesSQL, Process and Tooling at Scale
rocio
172
14k
Transcript
1 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBW %BUB%SJWFO8PSLqPX
https://github.com/triglav-dataflow
2 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 044 https://github.com/triglav-dataflow
˞υΩϡϝϯτগʑ͓ͪԼ͍͞
Importdb MySQL HDFS Vertica Medjed 01:00 06:00 Argus 10:00 3
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ❌Α͋͘ΔࠔΓ͝ͱ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ
Importdb MySQL HDFS Vertica Medjed 01:00 Argus Triglav Monitor Event
Monitor Event 4 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ⭕ରॲ๏ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ
5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Α͋͘ΔϫʔΫϑϩʔπʔϧͰμϝʁ •
શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ • ྫ શೖΓ+FOLJOTεϨʔϒ • ݹ ͍ʹ͑͠ ͷπʔϧͷଘࡏ • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍ • 31$ "1* Λ࣋ͨͳ͍ • +FOLJOTಉ࢜ͷ࿈ܞ ❌ࠔΓ͝ͱ %JHEBH -VJHJ +FOLJOTͳͲ৭ʑ͋Δ
5SJHMBWΞʔΩςΫνϟ ొਓ 6 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
DB agent consumer Triglav Core (Broker) 2. send message 1. polling message 1. watch every minute • .FTTBHF2VFVF • "HFOU 1SPEVDFS • ετϨʔδΛࢹ • $POTVNFS • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ • QVMMܕ • $PSF #SPLFS • ΠϕϯτΛཧ • QVMM͞Ε͙ͯ͢ʹΠϕϯτ ফ͑ͳ͍ σϑΥϧτอ࣋
HDFS Vertica BigQuery Triglav agent agent agent Medjed Jenkins Argus
Consumer Agent (Producer) Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBWΞʔΩςΫνϟ 7 • 5SJHMBWΛ)VCʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ • $POTVNFS5SJHMBWʹ͚ͩରԠ͢Εྑ͍
4XBHHFS 8 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
5SJHMBWͷ"1*༷Λ4XBHHFSͰهड़ • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB Λࣗಈੜ͍ͯ͠Δ RESTful APIͷ༷Λهड़͢Δ༷ͱͦͷπʔϧ܈
USJHMBWBHFOUIEGT KSVCZ 9 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• ࢹରͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ • ຖࢹ͢Δ • MTM3QBUIUP:NE • σϑΥϧτͰ • )BEPPQͳΒ*/PUJGZ"1*Λར༻͠ ͯΠϕϯτۦಈͰߋ৽ใΛऔΕΔ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕTFOENFTTBHF
USJHMBWBHFOUWFSUJDB DSVCZ 10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• جຊతʹUSJHMBWBHFOUIEGTͱಉ͡ • ςʔϒϧͷߦ͕͍࣋ͬͯΔFQPDIใΛར༻ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕύʔςΟγϣϯ୯ҐͰ·ͱΊ ͯTFOENFTTBHF
KFOLJOTUSJHHFSQPMMJOH@USJHMBW 11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
+FOLJOTϓϥάΠϯ • 5SJHMBWʹQPMMJOHͯ͠δϣϒ࣮ߦ
12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • .Z42-Ͱ.FTTBHF2VFVF࣮
ΧϥϜ໊ આ໌ ิ id ϝοηʔδID resource_uri σʔλϦιʔεͷURI ྫ) hdfs://path/to/%Y-%m-%d MQͷτϐ οΫʹ૬ resource_unit daily or hourly resource_timezone λΠϜκʔϯ ex) +09:00 resource_time Ϧιʔεߋ৽࣌ؒ 4&-&$5 GSPNNFTTBHFTXIFSFJE \PGGTFU^"/%SFTPVSDF@VSJ\VSJ^ 5SJHMBW࣮ৄࡉ
5SJHMBWύϑΥʔϚϯε 13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
USJHMBWBHFOU IEGTcWFSUJDB • ຖࢹ͢Δ͕ສఔͳ͠ • 4FOENFTTBHFසൟʹൃੜ͠ͳ͍ͨΊ ΄ͱΜͲؾʹͳΒͳ͍ • IPVSMZϦιʔεͰʹ௨ৗճ • ςʔϒϧ͋ͬͯ ఔ • 'FUDINFTTBHFຖߦΘΕΔ͕ɺ ΞϓϦαʔόNZTRMͷεϨʔϒΛ૿͢ ͳͲͰεέʔϧΞτ༰қ
'VUVSF8PSL 14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
ґଘͷՄࢹԽ • 'SPOU6*ͷ࡞ • ϩΰ • USJHMBWDPOTVNFSDMJ • Πϕϯτ͕དྷͨΒҾͷίϚϯυΛ࣮ߦ