Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
iQONを支えるデータ分析基盤/iqon-bigquery
Search
Masayuki Imamura
April 23, 2015
Programming
3
10k
iQONを支えるデータ分析基盤/iqon-bigquery
2015.04.23 えびスタ vol2での発表内容です @kyuns
Masayuki Imamura
April 23, 2015
Tweet
Share
More Decks by Masayuki Imamura
See All by Masayuki Imamura
バイセルにおけるAI活用の取り組みについて紹介します/Generative AI at BuySell Technologies
kyuns
2
710
経営視点から捉えた開発生産性 / Development productivity from a management perspective
kyuns
12
9.4k
Qiita:Teamをハックして成果をあげるための情報共有方法/Qiita:Team
kyuns
6
3.5k
3年連続ベストアプリ受賞のプロダクトを支える裏側/The way to Achieve The Best App 3 years in a row
kyuns
1
1.7k
機械学習とデータ分析を支えるマルチクラウドなアーキテクチャの紹介/Multi Cloud Architecture Supporting Machine Learning and Data Analysis
kyuns
4
9.8k
日本最大級のファッションDBを支える裏側/how to manage the complex web service
kyuns
4
860
iQONを支えるクローラー/iQON Crawler
kyuns
12
4.2k
iQON Tools
kyuns
1
3.9k
プッシュ通知大戦争/effective push notification by iQON
kyuns
28
8.4k
Other Decks in Programming
See All in Programming
バイラテラルアップサンプリング
fadis
3
430
マイコンでもRustのtestがしたい/KernelVM Kansai 11
tnishinaga
0
860
監視 やばい
syossan27
12
10k
はじめてのPDFKit.pdf
shomakato
0
100
エンジニアが挑む、限界までの越境
nealle
1
330
2025年のz-index設計を考える
tak_dcxi
6
3k
プロフェッショナルとしての成長「問題の深掘り」が導く真のスキルアップ / issue-analysis-and-skill-up
minodriven
8
1.9k
파급효과: From AI to Android Development
l2hyunwoo
0
160
複雑なフォームの jotai 設計 / Designing jotai(state) for Complex Forms #layerx_frontend
izumin5210
6
1.5k
20250426 GDGoC 合同新歓 - GDGoC のススメ
getty708
0
110
ComposeでのPicture in Picture
takathemax
0
130
開発者フレンドリーで顧客も満足?Platformの秘密
algoartis
0
200
Featured
See All Featured
Building Flexible Design Systems
yeseniaperezcruz
329
39k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.2k
Rails Girls Zürich Keynote
gr2m
94
13k
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
Building Applications with DynamoDB
mza
94
6.4k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
31
1.2k
Typedesign – Prime Four
hannesfritz
41
2.6k
What’s in a name? Adding method to the madness
productmarketing
PRO
22
3.4k
Automating Front-end Workflow
addyosmani
1370
200k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
49k
Code Review Best Practice
trishagee
68
18k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
Transcript
J20/Λࢧ͑Δ σʔλੳج൫ͱ#JH2VFSZ ͑ͼελWPM 7"4*-:!LZVOT
ࠓଜխ !LZVOT 7"4*-: *OD औక$50$P'PVOEFS ʹ:BIPP+"1"/ʹ৽ଔೖࣾ :BIPP'"4)*0/ɺ9#3"/%ͳͲͷϝσΟΞͷ্ཱͪ͛ ʹಠཱɺ7"4*-:Λۀɺऔక$50ʹब ͖ͳ༏ਫथಸʑ
ঁੑͷϑΝογϣϯϥΠϑΛΦγϟϨʹً͔ͤΔͨΊͷɺ ண͍ͨίʔσ͕ݟ͔ͭΔɺങ͑ΔɺஷΊΒΕΔϑΝογϣϯΞϓϦɻ ݱࡏສͷձһొ ٕज़ސ·ͭͱΏ͖ͻΖ
J20/ͷσʔλͱʁ
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ શͯ#JH2VFSZʹ͍Εͨ
ͳͥσʔλΛूΊͨͷ͔ w ࣾͷ৭ʑͳσʔλ͕༷ʑͳετϨʔδʹࢄ ͍ͯ͠Δʢ.Z42- .POHP%# ֤छ4BB4ʜ w σʔλΛՕॴʹूΊͯޮతʹੳ͍ͨ͠ w
#*πʔϧͰΰϦΰϦੳ͍ͨ͠ w μογϡϘʔυ࡞Γ͍ͨ
ͳͥ#JH2VFSZͳͷ͔ʁ w JNQPSUͷܗ͕ࣜॊೈͰ͋Δ w #*πʔϧରԠ UBCMFBV w ԯϨίʔυͰߴʹಈ͘ w
qVFOUEͷQMVHJO͕ἧͬͯΔ w TUSFBNJOHJOTFSUʹରԠ͍ͯ͠Δ
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ
ΞϓϦߦಈϩά ΞϓϦղੳπʔϧʮ-PDBMZUJDTʯͷੜσʔλ ϢʔβʔͷશΠϕϯτΛهͨ͠+40/ ԯߦ݄
ऩूΞʔΩςΫνϟ Amazon S3 "844 &$ (PPHMF$MPVE4UPSBHF #JH2VFSZ ΞϓϦ JNQPSUCBUDI -PDBMZUJDTͷTCVDLFU͔Β($4ܦ༝Ͱ#JH2VFSZʹDPQZ
ͷͰҰ(PPHMF$MPVE4UPSBHFʹDPQZͨ͠΄͏͕͍ ఆظతʹ#BUDIͰJNQPSU
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ
ϓογϡ௨ϩά w ΞϓϦͷϓογϡ௨ͷϩά w શͯͷϓογϡͷૹ৴ɺ։෧ϩά ສRQT w ϓογϡͷ$53ޮՌݕূλʔήςΟϯάΞϧ ΰϦζϜͳͲͷ࠷దԽʹར༻
w qVFOUE͕େ׆༂
ϩάϑΥʔϚοτ ͍ͭͲͷVTFSʹରͯ͠ͲΜͳϓογϡΛૹ͔ͬͨɺ։͍͔ͨ
ϓογϡϩάΞʔΩςΫνϟ #JH2VFSZ "1*4FSWFST qVFOUE "1/T ($. -PH4FSWFS qVFOUE qVFOUEܦ༝Ͱ#JH2VFSZʹ 4USFBNJOH*OTFSU
qVFOUQMVHJOCJHRVFSZΛར༻ ΞϓϦ Amazon S3 "844 ૹ৴ϩά ΫϦοΫϩά ૹ৴ˍΫϦοΫϩά
<case event.notification.click> <store> type bigquery method insert auth_method private_key email
[email protected]
private_key_path /home/vasily/fluentd/XXXXXX.p12 buffer_type file flush_interval 0 try_flush_interval 0.05 queued_chunk_flush_interval 0.01 buffer_chunk_records_limit 250 buffer_chunk_limit 512k bugger_queue_limit 1024 retry_limit 5 retry_wait 0.5 num_threads 32 dataset app project iqon-data-mining auto_create_table true table event_notification_click_%Y%m schema_path /home/vasily/fluentd/schema_event_notification_click.json buffer_path /var/log/td-agent/buffer/bigquery/event.notification.click </store> </case> ສRQTҎ্ʹ͑ΔͨΊʹ CV⒎FS@UZQFΛpMFʹ͠ͳ͍ͱ͍͚ͳ͔ͬͨ
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ
Ϛελʔσʔλ w .Z42- 3%4 ʹ͋ΔJ20/ͷதͷϚελʔσʔλ 6TFST *UFNT 4FUT #SBOET 4IPQʜFUD
w ϢʔβʔͷΞΫςΟϏςΟσʔλ ΞΠςϜ-*,& ίʔσ-*,& ϑΥϩʔʜFUD ԯϨίʔυͱ͔͋Δςʔϒϧୡ w "84্ͷ3%4ʹ֨ೲ͞Ε͍ͯΔͷΛ#JH2VFSZʹ 4ZOD
TZODํ๏ w ಠࣗεΫϦϓτͰҰׅJNQPSU w .Z42-ͷςʔϒϧఆ͔ٛΒࣗಈతʹ#JH2VFSZ ͷ4DIFNBΛੜ w ·Δ͝ͱςʔϒϧΛEVNQͯ͠·Δ͝ͱDPQZ w ઍສϨίʔυͰͦΜͳʹ͔͔࣌ؒΒͳ͍
ఔ
3%4UP#JH2VFSZ "843%4 &$ (PPHMF$MPVE4UPSBHF #JH2VFSZ JNQPSUCBUDI EFTDUBCMF@OBNFͨ͠༰͔Β#JH2VFSZͷTDIFNBKTPOΛੜ TFMFDU GSPNUBCMF@OBNFUBCMFUTWͨ͠༰Λ($4ʹVQMPBEͨ͠ޙɺ #JH2VFSZʹλϒ۠ΓUTWͱͯ͠DPQZΧϯϚ۠ΓͷσʔλʹରԠ͢ΔͨΊ
bq load --max_bad_record=1000 --project_id=iqon-data-mining —source_format=CSV --field_delimiter='\t' --skip_leading_rows=1 iqon-data- mining:app.#{@table_name} gs://iqon-rds/#{@table_name}.tsv schema_#{@table_name}.json
w ΞϓϦΠϕϯτϩά w ϓογϡ௨ϩά w Ϛελʔσʔλ w ηʔϧεσʔλ
ηʔϧεσʔλ w ΞΠςϜͷߪೖΫϦοΫɺߪೖྃσʔλ w ֨ೲઌ.POHP%# w ͱͯେ͖͍ίϨΫγϣϯʢे(
w NPOHPCRΛ༻͍ͯ#JH2VFSZʹҰׅJNQPSU !IBLPCFSB IUUQTHJUIVCDPNIBLPCFSBNPOHPCR
.POHP%#UP#JH2VFSZ &$ (PPHMF$MPVE4UPSBHF #JH2VFSZ JNQPSUCBUDI NPOHPCRίϚϯυΛ༻͍ͯTUSFBNͰॲཧͯ͘͠ΕΔ TDIFNBࣗಈఆػೳ͋Δ͕ɺಠࣗͰTDIFNBΛࢦఆͰ͖Δ ࢦఆ͢ΔLFZpMFQͰͳ͘KTPOܗࣜͷͷͳͷͰҙ mongobq --host
mongo02c --port 27017 --database iqon_conversion --collection click_log -q '{"date": "#{date}"}' --project iqon-data-mining --dataset app --keyfile /home/vasily/XXXX.json -B iqon-mongo -T click_log_test --schema ./ schema_mongo_iqon_conversion_click_log.json --autoclean
ΞΫηεσʔλ ൪֎ฤ w (PPHMF"OBMZUJDTͷσʔλ (PPHMF"OBMZUJDTϓϨϛΞϜͳΒ#JH2VFSZʹ ੜσʔλΛࣗಈతʹJNQPSUͯ͘͠ΕΔ w ສԁ݄͙Β͍ͱ͍͏ᷚ
·ͱΊ w શͯͷσʔλϩάΛ#JH2VFSZՕॴʹूΊͨ ͓͔͛Ͱੳ͕֨ஈʹޮతʹͳͬͨ ϚελʔσʔλͱϩάσʔλΛ+0*/ͯ͠ੳ w #JH2VFSZ҆ͯ͘͏·͍ 5#ετϨʔδ݄ͬͯ ԁఔ εΩϟϯྉ͕5#͋ͨΓ݄ԁఔ
ݱࡏ
8F`SF)JSJOH J20/ຊͰҰ൪ϑΝογϣϯʹ ؔ͢Δσʔλ͕ू·ΔॴͰ͢ɻ σʔλΛͬͯνϟϨϯδͯ͠Έ͍ͨਓΛ ͓͓ͪͯ͠Γ·͢ɻ JOGP!WBTJMZKQ