Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Triglav - Data Driven Workflow Tool
Search
Naotoshi Seo
February 17, 2017
1
4.1k
Triglav - Data Driven Workflow Tool
Naotoshi Seo
February 17, 2017
Tweet
Share
More Decks by Naotoshi Seo
See All by Naotoshi Seo
ZOZOTOWNリプレイス2020
sonots
5
36k
Red Chainer and Cumo: Practical Deep Learning in Ruby at RubyKaigi 2019
sonots
1
4.4k
Introduction of Cumo, and Integration to Red Chainer
sonots
1
1.1k
Implementation of Cumo, a CUDA-aware version of Ruby/Numo
sonots
1
1.9k
Fast Numerical Computing and Deep Learning in Ruby with Cumo
sonots
0
9.3k
CuPy improvments around memory
sonots
3
1.6k
DeNA AIシステム部におけるクラウドを活用した機械学習基盤の構築
sonots
4
6.2k
DeNA流データエンジニアリングの極意
sonots
17
13k
BigQuery Schema Migration #bq_sushi
sonots
2
6.1k
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
280
13k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
7
630
Six Lessons from altMBA
skipperchong
27
3.7k
How to Think Like a Performance Engineer
csswizardry
22
1.5k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
28
9.4k
Designing Experiences People Love
moore
141
23k
Stop Working from a Prison Cell
hatefulcrawdad
268
20k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
118
51k
The Art of Programming - Codeland 2020
erikaheidi
53
13k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
102
19k
Site-Speed That Sticks
csswizardry
4
470
Transcript
1 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBW %BUB%SJWFO8PSLqPX
https://github.com/triglav-dataflow
2 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 044 https://github.com/triglav-dataflow
˞υΩϡϝϯτগʑ͓ͪԼ͍͞
Importdb MySQL HDFS Vertica Medjed 01:00 06:00 Argus 10:00 3
Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ❌Α͋͘ΔࠔΓ͝ͱ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ • ࣌ؒࢦఆͰσʔλϑϩʔʹͭͳ͕ΓΛ͍࣋ͨͤͯΔ • ࠶࣮ߦͨ࣌͠ʹ࿈ܞ͞Εͳ͍ ɾТɾʆ
Importdb MySQL HDFS Vertica Medjed 01:00 Argus Triglav Monitor Event
Monitor Event 4 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. τϦάϥϑͱ ⭕ରॲ๏ • σʔλۦಈͰͭͳ͕Γͷͳ͍πʔϧʹͭͳ͕ΓΛ࣋ ͨͤͯϫʔΫϑϩʔܗΛࢧԉ͢Δπʔϧ
5 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. Α͋͘ΔϫʔΫϑϩʔπʔϧͰμϝʁ •
શͯͷπʔϧΛ࣮ߦϗετͰ࣮ߦͰ͖Δલఏ • ྫ શೖΓ+FOLJOTεϨʔϒ • ݹ ͍ʹ͑͠ ͷπʔϧͷଘࡏ • ࣮ߦϗετΛҠͤͳ͍ Ҡͨ͘͠ͳ͍ • 31$ "1* Λ࣋ͨͳ͍ • +FOLJOTಉ࢜ͷ࿈ܞ ❌ࠔΓ͝ͱ %JHEBH -VJHJ +FOLJOTͳͲ৭ʑ͋Δ
5SJHMBWΞʔΩςΫνϟ ొਓ 6 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
DB agent consumer Triglav Core (Broker) 2. send message 1. polling message 1. watch every minute • .FTTBHF2VFVF • "HFOU 1SPEVDFS • ετϨʔδΛࢹ • $POTVNFS • ΠϕϯτΛऔಘͯ͠δϣϒ࣮ߦ • QVMMܕ • $PSF #SPLFS • ΠϕϯτΛཧ • QVMM͞Ε͙ͯ͢ʹΠϕϯτ ফ͑ͳ͍ σϑΥϧτอ࣋
HDFS Vertica BigQuery Triglav agent agent agent Medjed Jenkins Argus
Consumer Agent (Producer) Copyright (C) DeNA Co.,Ltd. All Rights Reserved. 5SJHMBWΞʔΩςΫνϟ 7 • 5SJHMBWΛ)VCʹͨ͠ΠϕϯτۦಈΞʔΩςΫνϟ • $POTVNFS5SJHMBWʹ͚ͩରԠ͢Εྑ͍
4XBHHFS 8 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
5SJHMBWͷ"1*༷Λ4XBHHFSͰهड़ • USJHMBWDMJFOUSVCZɺUSJHMBWDMJFOUKBWB Λࣗಈੜ͍ͯ͠Δ RESTful APIͷ༷Λهड़͢Δ༷ͱͦͷπʔϧ܈
USJHMBWBHFOUIEGT KSVCZ 9 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• ࢹରͷ)%'4ύεΛ5SJHMBWίΞ͔Βऔಘ • ຖࢹ͢Δ • MTM3QBUIUP:NE • σϑΥϧτͰ • )BEPPQͳΒ*/PUJGZ"1*Λར༻͠ ͯΠϕϯτۦಈͰߋ৽ใΛऔΕΔ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕTFOENFTTBHF
USJHMBWBHFOUWFSUJDB DSVCZ 10 Copyright (C) DeNA Co.,Ltd. All Rights Reserved.
• جຊతʹUSJHMBWBHFOUIEGTͱಉ͡ • ςʔϒϧͷߦ͕͍࣋ͬͯΔFQPDIใΛར༻ • લճͷ࠷৽࣌ؒͱߋ৽࣌ؒΛൺֱͯ͠ɺ৽ͨʹσʔ λ͕ߋ৽͞Ε͍ͯΕύʔςΟγϣϯ୯ҐͰ·ͱΊ ͯTFOENFTTBHF
KFOLJOTUSJHHFSQPMMJOH@USJHMBW 11 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
+FOLJOTϓϥάΠϯ • 5SJHMBWʹQPMMJOHͯ͠δϣϒ࣮ߦ
12 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. • .Z42-Ͱ.FTTBHF2VFVF࣮
ΧϥϜ໊ આ໌ ิ id ϝοηʔδID resource_uri σʔλϦιʔεͷURI ྫ) hdfs://path/to/%Y-%m-%d MQͷτϐ οΫʹ૬ resource_unit daily or hourly resource_timezone λΠϜκʔϯ ex) +09:00 resource_time Ϧιʔεߋ৽࣌ؒ 4&-&$5 GSPNNFTTBHFTXIFSFJE \PGGTFU^"/%SFTPVSDF@VSJ\VSJ^ 5SJHMBW࣮ৄࡉ
5SJHMBWύϑΥʔϚϯε 13 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
USJHMBWBHFOU IEGTcWFSUJDB • ຖࢹ͢Δ͕ສఔͳ͠ • 4FOENFTTBHFසൟʹൃੜ͠ͳ͍ͨΊ ΄ͱΜͲؾʹͳΒͳ͍ • IPVSMZϦιʔεͰʹ௨ৗճ • ςʔϒϧ͋ͬͯ ఔ • 'FUDINFTTBHFຖߦΘΕΔ͕ɺ ΞϓϦαʔόNZTRMͷεϨʔϒΛ૿͢ ͳͲͰεέʔϧΞτ༰қ
'VUVSF8PSL 14 Copyright (C) DeNA Co.,Ltd. All Rights Reserved. •
ґଘͷՄࢹԽ • 'SPOU6*ͷ࡞ • ϩΰ • USJHMBWDPOTVNFSDMJ • Πϕϯτ͕དྷͨΒҾͷίϚϯυΛ࣮ߦ