Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
iQONを支えるデータ分析基盤/iqon-bigquery
Search
Masayuki Imamura
April 23, 2015
Programming
3
10k
iQONを支えるデータ分析基盤/iqon-bigquery
2015.04.23 えびスタ vol2での発表内容です @kyuns
Masayuki Imamura
April 23, 2015
Tweet
Share
More Decks by Masayuki Imamura
See All by Masayuki Imamura
Qiita:Teamをハックして成果をあげるための情報共有方法/Qiita:Team
kyuns
6
3.1k
3年連続ベストアプリ受賞のプロダクトを支える裏側/The way to Achieve The Best App 3 years in a row
kyuns
1
1.5k
機械学習とデータ分析を支えるマルチクラウドなアーキテクチャの紹介/Multi Cloud Architecture Supporting Machine Learning and Data Analysis
kyuns
4
9.3k
日本最大級のファッションDBを支える裏側/how to manage the complex web service
kyuns
4
770
iQONを支えるクローラー/iQON Crawler
kyuns
12
4k
iQON Tools
kyuns
1
3.8k
プッシュ通知大戦争/effective push notification by iQON
kyuns
28
8.1k
VASILY流エンジニアドリブン / vasily engineer driven way
kyuns
2
2.2k
iQONのグロースハックを支えるチームと実践手法
kyuns
5
3.8k
Other Decks in Programming
See All in Programming
VSCodeでのDatabricks開発もお勧めしたい/I would also recommend Databricks development with VSCode.
kazumain
0
250
try!Swift Tokyo 2024 参加報告 LT
akidon0000
1
220
Goのエラースタックトレースの歴史と今後
sonatard
8
1.4k
スキーマ駆動開発による品質とスピードの両立 - 私達は何故、スキーマを書くのか
kentaroutakeda
0
170
0→1と1→10の狭間で Javaという技術選定を振り返る/Reflecting on the Decision to Choose Java Between Scaling from 0 to 1 and 1 to 10
jaguar_imo
2
380
[技育CAMPアカデミア]アイディアを形に!【超入門】スマホアプリ開発〜リリースまでの流れをご紹介
teamlab
PRO
0
370
Java 22 Overview
kishida
1
180
Git Rebase
bkuhlmann
11
1.6k
Zero Waste, Radical Magic, and Italian Graft – Quarkus Efficiency Secrets
hollycummins
0
230
2 週間で Twitter Bot を作ってみた
contour_gara
0
380
Blue/Greenデプロイの導入による 運用フローの改善
kudoas
1
370
サイコロで理解する統計的仮説検定の考え方
tatamiya
4
930
Featured
See All Featured
A Philosophy of Restraint
colly
197
16k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
19
1.7k
Code Reviewing Like a Champion
maltzj
514
39k
What the flash - Photography Introduction
edds
64
11k
[RailsConf 2023] Rails as a piece of cake
palkan
23
3.9k
Building Effective Engineering Teams - LeadDev
addyosmani
28
1.8k
The Pragmatic Product Professional
lauravandoore
25
5.8k
jQuery: Nuts, Bolts and Bling
dougneiner
59
7.1k
Large-scale JavaScript Application Architecture
addyosmani
504
110k
How To Stay Up To Date on Web Technology
chriscoyier
782
250k
Visualization
eitanlees
136
14k
Adopting Sorbet at Scale
ufuk
68
8.6k
Transcript
J20/Λࢧ͑Δ σʔλੳج൫ͱ#JH2VFSZ ͑ͼελWPM 7"4*-:!LZVOT
ࠓଜխ !LZVOT 7"4*-: *OD औక$50$P'PVOEFS ʹ:BIPP+"1"/ʹ৽ଔೖࣾ :BIPP'"4)*0/ɺ9#3"/%ͳͲͷϝσΟΞͷ্ཱͪ͛ ʹಠཱɺ7"4*-:Λۀɺऔక$50ʹब ͖ͳ༏ਫथಸʑ
ঁੑͷϑΝογϣϯϥΠϑΛΦγϟϨʹً͔ͤΔͨΊͷɺ ண͍ͨίʔσ͕ݟ͔ͭΔɺങ͑ΔɺஷΊΒΕΔϑΝογϣϯΞϓϦɻ ݱࡏສͷձһొ ٕज़ސ·ͭͱΏ͖ͻΖ
J20/ͷσʔλͱʁ
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ શͯ#JH2VFSZʹ͍Εͨ
ͳͥσʔλΛूΊͨͷ͔ w ࣾͷ৭ʑͳσʔλ͕༷ʑͳετϨʔδʹࢄ ͍ͯ͠Δʢ.Z42- .POHP%# ֤छ4BB4ʜ w σʔλΛՕॴʹूΊͯޮతʹੳ͍ͨ͠ w
#*πʔϧͰΰϦΰϦੳ͍ͨ͠ w μογϡϘʔυ࡞Γ͍ͨ
ͳͥ#JH2VFSZͳͷ͔ʁ w JNQPSUͷܗ͕ࣜॊೈͰ͋Δ w #*πʔϧରԠ UBCMFBV w ԯϨίʔυͰߴʹಈ͘ w
qVFOUEͷQMVHJO͕ἧͬͯΔ w TUSFBNJOHJOTFSUʹରԠ͍ͯ͠Δ
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ
ΞϓϦߦಈϩά ΞϓϦղੳπʔϧʮ-PDBMZUJDTʯͷੜσʔλ ϢʔβʔͷશΠϕϯτΛهͨ͠+40/ ԯߦ݄
ऩूΞʔΩςΫνϟ Amazon S3 "844 &$ (PPHMF$MPVE4UPSBHF #JH2VFSZ ΞϓϦ JNQPSUCBUDI -PDBMZUJDTͷTCVDLFU͔Β($4ܦ༝Ͱ#JH2VFSZʹDPQZ
ͷͰҰ(PPHMF$MPVE4UPSBHFʹDPQZͨ͠΄͏͕͍ ఆظతʹ#BUDIͰJNQPSU
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ
ϓογϡ௨ϩά w ΞϓϦͷϓογϡ௨ͷϩά w શͯͷϓογϡͷૹ৴ɺ։෧ϩά ສRQT w ϓογϡͷ$53ޮՌݕূλʔήςΟϯάΞϧ ΰϦζϜͳͲͷ࠷దԽʹར༻
w qVFOUE͕େ׆༂
ϩάϑΥʔϚοτ ͍ͭͲͷVTFSʹରͯ͠ͲΜͳϓογϡΛૹ͔ͬͨɺ։͍͔ͨ
ϓογϡϩάΞʔΩςΫνϟ #JH2VFSZ "1*4FSWFST qVFOUE "1/T ($. -PH4FSWFS qVFOUE qVFOUEܦ༝Ͱ#JH2VFSZʹ 4USFBNJOH*OTFSU
qVFOUQMVHJOCJHRVFSZΛར༻ ΞϓϦ Amazon S3 "844 ૹ৴ϩά ΫϦοΫϩά ૹ৴ˍΫϦοΫϩά
<case event.notification.click> <store> type bigquery method insert auth_method private_key email
[email protected]
private_key_path /home/vasily/fluentd/XXXXXX.p12 buffer_type file flush_interval 0 try_flush_interval 0.05 queued_chunk_flush_interval 0.01 buffer_chunk_records_limit 250 buffer_chunk_limit 512k bugger_queue_limit 1024 retry_limit 5 retry_wait 0.5 num_threads 32 dataset app project iqon-data-mining auto_create_table true table event_notification_click_%Y%m schema_path /home/vasily/fluentd/schema_event_notification_click.json buffer_path /var/log/td-agent/buffer/bigquery/event.notification.click </store> </case> ສRQTҎ্ʹ͑ΔͨΊʹ CV⒎FS@UZQFΛpMFʹ͠ͳ͍ͱ͍͚ͳ͔ͬͨ
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ
Ϛελʔσʔλ w .Z42- 3%4 ʹ͋ΔJ20/ͷதͷϚελʔσʔλ 6TFST *UFNT 4FUT #SBOET 4IPQʜFUD
w ϢʔβʔͷΞΫςΟϏςΟσʔλ ΞΠςϜ-*,& ίʔσ-*,& ϑΥϩʔʜFUD ԯϨίʔυͱ͔͋Δςʔϒϧୡ w "84্ͷ3%4ʹ֨ೲ͞Ε͍ͯΔͷΛ#JH2VFSZʹ 4ZOD
TZODํ๏ w ಠࣗεΫϦϓτͰҰׅJNQPSU w .Z42-ͷςʔϒϧఆ͔ٛΒࣗಈతʹ#JH2VFSZ ͷ4DIFNBΛੜ w ·Δ͝ͱςʔϒϧΛEVNQͯ͠·Δ͝ͱDPQZ w ઍສϨίʔυͰͦΜͳʹ͔͔࣌ؒΒͳ͍
ఔ
3%4UP#JH2VFSZ "843%4 &$ (PPHMF$MPVE4UPSBHF #JH2VFSZ JNQPSUCBUDI EFTDUBCMF@OBNFͨ͠༰͔Β#JH2VFSZͷTDIFNBKTPOΛੜ TFMFDU GSPNUBCMF@OBNFUBCMFUTWͨ͠༰Λ($4ʹVQMPBEͨ͠ޙɺ #JH2VFSZʹλϒ۠ΓUTWͱͯ͠DPQZΧϯϚ۠ΓͷσʔλʹରԠ͢ΔͨΊ
bq load --max_bad_record=1000 --project_id=iqon-data-mining —source_format=CSV --field_delimiter='\t' --skip_leading_rows=1 iqon-data- mining:app.#{@table_name} gs://iqon-rds/#{@table_name}.tsv schema_#{@table_name}.json
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ
ηʔϧεσʔλ w ΞΠςϜͷߪೖΫϦοΫɺߪೖྃσʔλ w ֨ೲઌ.POHP%# w ͱͯେ͖͍ίϨΫγϣϯʢे(
w NPOHPCRΛ༻͍ͯ#JH2VFSZʹҰׅJNQPSU !IBLPCFSB IUUQTHJUIVCDPNIBLPCFSBNPOHPCR
.POHP%#UP#JH2VFSZ &$ (PPHMF$MPVE4UPSBHF #JH2VFSZ JNQPSUCBUDI NPOHPCRίϚϯυΛ༻͍ͯTUSFBNͰॲཧͯ͘͠ΕΔ TDIFNBࣗಈఆػೳ͋Δ͕ɺಠࣗͰTDIFNBΛࢦఆͰ͖Δ ࢦఆ͢ΔLFZpMFQͰͳ͘KTPOܗࣜͷͷͳͷͰҙ mongobq --host
mongo02c --port 27017 --database iqon_conversion --collection click_log -q '{"date": "#{date}"}' --project iqon-data-mining --dataset app --keyfile /home/vasily/XXXX.json -B iqon-mongo -T click_log_test --schema ./ schema_mongo_iqon_conversion_click_log.json --autoclean
ΞΫηεσʔλ ൪֎ฤ w (PPHMF"OBMZUJDTͷσʔλ (PPHMF"OBMZUJDTϓϨϛΞϜͳΒ#JH2VFSZʹ ੜσʔλΛࣗಈతʹJNQPSUͯ͘͠ΕΔ w ສԁ݄͙Β͍ͱ͍͏ᷚ
·ͱΊ w શͯͷσʔλϩάΛ#JH2VFSZՕॴʹूΊͨ ͓͔͛Ͱੳ͕֨ஈʹޮతʹͳͬͨ ϚελʔσʔλͱϩάσʔλΛ+0*/ͯ͠ੳ w #JH2VFSZ҆ͯ͘͏·͍ 5#ετϨʔδ݄ͬͯ ԁఔ εΩϟϯྉ͕5#͋ͨΓ݄ԁఔ
ݱࡏ
8F`SF)JSJOH J20/ຊͰҰ൪ϑΝογϣϯʹ ؔ͢Δσʔλ͕ू·ΔॴͰ͢ɻ σʔλΛͬͯνϟϨϯδͯ͠Έ͍ͨਓΛ ͓͓ͪͯ͠Γ·͢ɻ JOGP!WBTJMZKQ