Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Google BigQuery の話 #gcpja
Search
Naoya Ito
September 17, 2014
Technology
17
5.7k
Google BigQuery の話 #gcpja
gcp ja night で話した BigQuery のスライド。YAPC::Asia のものに数枚だけスライドを追加したもので、ほぼ同じです。
Naoya Ito
September 17, 2014
Tweet
Share
More Decks by Naoya Ito
See All by Naoya Ito
Haskell でアルゴリズムを抽象化する / 関数型言語で競技プログラミング
naoya
17
4.9k
Functional TypeScript
naoya
15
6.2k
TypeScript 関数型スタイルでバックエンド開発のリアル
naoya
71
34k
シェルの履歴とイクンリメンタル検索を使う
naoya
8
3.2k
20230227-engineer-type-talk.pdf
naoya
88
75k
関数型プログラミングと型システムのメンタルモデル
naoya
62
100k
TypeScript による GraphQL バックエンド開発
naoya
28
35k
フロントエンドのパラダイムを参考にバックエンド開発を再考する / TypeScript による GraphQL バックエンド開発
naoya
67
24k
「問題から目を背けず取り組む」 一休の開発チームが6年間で学んだこと
naoya
144
59k
Other Decks in Technology
See All in Technology
BrainPadプログラミングコンテスト記念LT会2025_社内イベント&問題解説
brainpadpr
1
160
ローカルLLMでファインチューニング
knishioka
0
150
AIのAIによるAIのための出力評価と改善
chocoyama
2
540
20250623 Findy Lunch LT Brown
3150
0
850
解析の定理証明実践@Lean 4
dec9ue
0
170
Amazon ECS & AWS Fargate 運用アーキテクチャ2025 / Amazon ECS and AWS Fargate Ops Architecture 2025
iselegant
16
5.4k
あなたの声を届けよう! 女性エンジニア登壇の意義とアウトプット実践ガイド #wttjp / Call for Your Voice
kondoyuko
4
410
CSS、JSをHTMLテンプレートにまとめるフロントエンド戦略
d120145
0
280
ひとり情シスなCTOがLLMと始めるオペレーション最適化 / CTO's LLM-Powered Ops
yamitzky
0
420
地図も、未来も、オープンに。 〜OSGeo.JPとFOSS4Gのご紹介〜
wata909
0
110
Fabric + Databricks 2025.6 の最新情報ピックアップ
ryomaru0825
1
130
5min GuardDuty Extended Threat Detection EKS
takakuni
0
130
Featured
See All Featured
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
790
Product Roadmaps are Hard
iamctodd
PRO
54
11k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
138
34k
Writing Fast Ruby
sferik
628
61k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
228
22k
RailsConf 2023
tenderlove
30
1.1k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Building Flexible Design Systems
yeseniaperezcruz
328
39k
Done Done
chrislema
184
16k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
8
670
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.5k
Transcript
(PPHMF#JH2VFSZͷ /BPZB*UP ,"*;&/QMBUGPSN*OD HDQKBOJHIU
ΞδΣϯμ • #JH2VFSZ֓؍ • #JH2VFSZͷ෦ • ,"*;&/QMBUGPSN*ODͰͷ͍Ͳ͜Ζ
#JH2VFSZ֓؍
(PPHMF#JH2VFSZ
None
#JH2VFSZͱ • ڊେͳσʔλͷ42- ͳͲ ΛඵͰ࣮ߦ͢ΔΫϥυαʔϏε – ԯϨίʔυΛඵ ˞ –
8FCΠϯλʔϑΣʔε͓Αͼ3&45"1* • (PPHMFࣾͰΘΕ͖ͯͨ%SFNFMΛαʔϏεԽ – ݄$MPTFEϦϦʔε – ݄Ұൠެ։ – ܧଓతʹόʔδϣϯΞοϓ – ݄#JH2VFSZ4USFBNJOH ˞(PPHMFͷދͷࢠʮ#JH2VFSZʯΛ'MVFOUEϢʔβʔ͕Θͳ͍ཧ༝͕ͳ͘ͳͬͨཧ༝ IUUQRJJUBDPNLB[VOPSJJUFNTBDBDCCBBBG
ͲΜͳ͜ͱʹΘΕΔ͔ • Ϣʔεέʔε – ϩάղੳ – %BUBXBSF)PVTF – • ͍ͯͳ͍༻్ – ۀ%# ͍3%#.4Ͱ
ͳ͍Αɺͱ͍͏͜ͱ
#JH2VFSZͳ͍͔ͥ • جຊɺϑϧεΩϟϯͰ͕ΜΔ – 3%#.4ͷ#5SFFΠϯσοΫεͱ͔ͳ͍ • 42-Λࢄॲཧ – .11 .BTTJWFMZ1BSBMMFM1SPDFTTJOH
2VFSZ&OHJOF %SFNFM • ઍͷσΟεΫͱߴωοτϫʔΫͰεέʔϧΞτ – 5#ͷσʔλΛඵͰϦʔυ͢Δ*0
ͨͩ͠ • ͍3%#.4Ͱͳ͍ • େਓͰҰʹ͏ͷͰͳ͍ – ओʹόονॲཧʹ͏ • εΩʔϚϨεͰͳ͍ 5#نσʔλͰઢܗҎ ԼͰεέʔϧ͢Δ͕ɺٯ
ʹখ͞ͳσʔλͰඵ ͷΦʔόʔϔου͕͋Δ ͷͰ
BigQuery読書会、@harukasan 資料より引用
ଞͷྨࣅ࣮ͱͷϙδγϣχϯά • -BSHF#BUDI – ҆ఆͯ͠ڊେͳόονΛ࣮ߦͰ͖Δ – ΫΤϦ࣮ߦ࣌ͷΦʔόʔϔου͕େ͖͍ ेඵʙे –
.BQ3FEVDFɺ)BEPPQ )JWF • 4IPSU#BUDI – ΫΤϦ࣮ߦ࣌ͷΦʔόʔϔου͕NTʙඵ – ΞυϗοΫΫΤϦʹ͍͍ͯΔ – .112VFSZ&OHJOF1SFTUPɺ*NQBMBɺ#JH2VFSZ %SFNFM • 4USFBN1SPDFTTJOH – όον࣮ߦͰ͖ͳ͍͕ετϦʔϜʹରͯ͠ϦΞϧλΠϜॲཧͰ͖Δ – /PSJLSBɺ"QBDIF,BGLBɺ5XJUUFS4UPSNFUD "NB[PO3FETIJGU 4IPSU#BUDI ৄ͘͠ ͳ͍ͷͰলུ cf. Batch processing and Stream processing by SQL h;p://www.slideshare.net/tagomoris/hcj2014-‐sql
Ձ֨ • ྉۚ – σʔλอ(#݄ – ΫΤϦ5# εΩϟϯͨ͠σʔλͷαΠ ζ "NB[PO4ΑΓ࣮
͍҆ νέοτΒ͍·ͨ͠
#JH2VFSZͷ෦ ͚ͩ͢͜͠
(PPHMF#JH%BUB4UBDL • ʰ(PPHMFΛࢧ͑Δٕज़ʱ – #JH%BUB4UBDL – ('4ɺ#JH5BCMFɺ.BQ3FEVDFFUD • #JH%BUB4UBDL –
#JH%BUB4UBDLͷ্ʹߏங͞Εͨɺͷ՝Λղফ͢Δ࣮܈ – $PMPTTVT .FHBTUPSF 4QBOOFS 'MVNF+BWB %SFNFM طʹ(PPHMFࣾ #JH%BUB4UBDLͩ ͱ͔͍͏ͪΒ΄Β
#JH2VFSZͷٕज़ελοΫ (PPHMF'JMF4ZTUFN ('4 $PMPTTVT'JMF4ZTUFN $'4 $PMVNO*0 %SFNFM ࢄ'4
('4ͷվྑܕ'4 ৄࡉඇެ։ #JH2VFSZͷͨΊͷྻࢦϑΝΠϧ ϑΥʔϚοτ ฒྻ42-࣮ߦΤϯδϯ σʔληϯλʔΛ·͍ͨͰ ࢄ͞ΕͯΔσʔλΛฒྻ ͔ͭߴʹऔಘͰ͖ΔΒ͠ ͍
$PMVNO*0 Dremel: InteracIve Analysis of Web-‐Scale Datasets h;p://research.google.com/pubs/archive/36632.pdf ߦͰͳ͘ྻ୯ҐͰɻಛ
ఆྻΛγʔέϯγϟϧʹ ಡΊΔͭ$PMPTTVT ͰฒྻಡΈࠐΈ
%SFNFM Dremel: InteracIve Analysis of Web-‐Scale Datasets h;p://research.google.com/pubs/archive/36632.pdf
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon ࢄ
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ࢄ ू
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ྻΛॱ൪ʹಡΈߦ Λऔಘɻ8)&3&۟ͳ ͲΛݟͯඞཁͳߦͷΈ ʹߜΓϝϞϦͰอ࣋ ࢄ ू
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ྻΛॱ൪ʹಡΈߦ Λऔಘɻ8)&3&۟ͳ ͲΛݟͯඞཁͳߦͷΈ ʹߜΓϝϞϦͰอ࣋ ֤TIBSE͔ΒσʔλΛू ɻྫ͑ιʔτ-*.*5 ͷߜΓࠐΈͳͲ͢Δ ࢄ ू
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ྻΛॱ൪ʹಡΈߦ Λऔಘɻ8)&3&۟ͳ ͲΛݟͯඞཁͳߦͷΈ ʹߜΓϝϞϦͰอ࣋ ֤TIBSE͔ΒσʔλΛू ɻྫ͑ιʔτ-*.*5 ͷߜΓࠐΈͳͲ͢Δ ूͨ݁͠Ռ ΛDBMMFSʹฦ͢ ࢄ ू
#JH2VFSZͷ͍͢͝ॴ • ΧϥϜܕ*0ɺ42-ͷׂ౷࣏ – Ͱ͜Εɺ.11తʹ͘͠ͳ͍ • ͡Ό͋ɺ#JH2VFSZͷԿ͕͍͔͢͝ – (PPHMFͷͰ͔͍Πϯϑϥ
ׂͱ֖ͳ͍ŋŋŋ
͜ΜͳΫιΫΤϦͰඵɺ̐ඵͩ
,"*;&/QMBUGPSN*OD Ͱͷ͍Ͳ͜Ζ
Ϣʔεέʔε • ΞΫηεϩάͷอଘௐࠪ • ΞϓϦέʔγϣϯϩάͷղੳ %BUBXBSF )PVTF • "#ςετͷ༗ҙࠩఆ
ΞΫηεϩά
ΞΫηεϩά #JH2VFSZ • /HJOYͷϩάΛqVFOUQMVHJOCJHRVFSZͰ ૹΓଓ͚Δ – &&Ͱ҉߸Խ͞ΕͯΔΑ • Կ͔༻͕͋ͬͨΒ42-Ͱղੳ –
%BJMZ8FFLMZ.POUIMZ17 – ϓϩμΫγϣϯͷσόοά
qVFOUQMVHJOCJHRVFSZ • CZUBHPNPSJT͞ΜɺZVHVJ͞Μଞ • ઌ͔Β,"*;&/QMBUGPSN*OD͕ϝ ϯςφʹ – ࣮࣭ɺԶ QBUDIFTXFMDPNF Ͱ͢
ΞϓϦέʔγϣϯͷϩάղੳ
ϩάΛඈ͢ • 3BJMT͔ΒUEMPHHFSSVCZͰqVFOUE • qVFOUEQMVHJOCJHRVFSZͰ#2ʹඈ͢
ϩάΛඈ͢ܖػ • ϦΫΤετຖ – "QQMJDBUJPO$POUSPMMFS – ϩάΠϯϢʔβͷଐੑΛඈ͢ˠ%"6."6ͷ ࢉग़ʹ • Ϟσϧͷঢ়ଶมߋ࣌
– "DUJWF3FDPSE0CTFSWFS – ϞσϧຖʹదͳଐੑΛݟસͬͯඈ͢ – #JH2VFSZෳࡶͳ42-Ͱී௨ʹԠ͢Δ㱺ϓ ϩμΫτϚωʔδϟ͕ؾܰʹ42-ॻ͍ͯΔ
ਖ਼نԽ͋·Γ͠ͳ͍ • ελʔεΩʔϚ – %8)ͷఆ൪ͷϞσϦϯά • ϑΝΫτςʔϒϧŋŋŋϩά • ࣍ݩςʔϒϧŋŋŋϚελʔσʔλ ސ٬໊ͱ͔
– ਖ਼نԽ͠ͳ͍ͷ͕ηΦϦʔ
"#ςετ༗ҙࠩఆ • "#ςετͷαʔϏεͳͷͰ͆ • ৄࡉൿີ • SFRTFDͱ͔qVUFOEͰૹͬͯΔ ͚ͲͬͪΌΒ͞ – ˞SFRTFDͷ)551SFRVFTUqVFOUE͕όοϑΝϦϯά͢ΔͷͰ
#JH2VFSZͷ"1*ίʔϧͣͬͱগͳ͍
֎෦πʔϧͱͷଓ • ΤΫηϧ – #JH2VFSZ$POOFDUPSGPS&YDFMCZ(PPHMF – ϐϘοτੳʹ • %0.0 #*
– FYQFSJNFOUBMͳ#JH2VFSZΠϯλϑΣʔε ͋ͬͨ – 5BCMFBVϝδϟʔͲ͜ΖରԠ࢝͠ΊͯΔ
໘ͳͱ͜Ζ • qVFOUEQMVHJOCJHRVFSZ͕εΩʔϚϑΝΠϧΛཁٻ ͢Δ – ͕͔ͩ͠͠IBLPCFSB͞Μ͕QBUDIΛॻ͍ͯ͘Εͨ – W͔ΒGFUDI@TDIFNBػೳ͕͑ΔΑ • ࣍ݩςʔϒϧͷߋ৽
– 61%"5&Ͱ͖ͳ͍ͷͰ – ؒͱ͔ʹҰճফͯ͠࡞ΔɺΈ͍ͨͳ – 1SFTUPΈ͍ͨʹҧ͏σʔλιʔεΛ+0*/Ͱ͖ͨΓ͢Δͱخ ͍͠ͷ͕ͩŋŋŋ
࢛ํࢁͦͷ • 42-ͱ͍ͬͯඪ४42-͡Όͳ͍Α – 3&(&91@."5$) ͱ͔3&(&91@&953"$5 ͱ͔+40/ ͱ ͔501 ͱ͔
• ʮͲ͏ͤϑϧεΩϟϯͯ͠Δ͠ʯͱ͍͏લఏʹཱͭͱΑ ͍ – -&'5 '03."5@65$@64&$ UJNF BTEBZ (3061#:EBZͱ͔ – 3&(&91@&953"$5 UJUMF S aX BTGSBHNFOU(3061#: GSBHNFOU03%&3#:GSBHNFOU@DPVOUEFTDͱ͔ – αϒΫΤϦ7JFX
࢛ํࢁͦͷ • 61%"5&%&-&5&ͳ͍ – ཁΒͳ͍ΧϥϜʹOVMM • ΧϥϜܕ͔ͩΒOVMMͳΒ༰ྔ৯Θͳ͍ – εΩʔϚՃ؆୯ • ߋ৽جຊআͯ͠࡞Γ͠
࢛ํࢁͦͷ • (PPHMF"OBMZUJDT #JH2VFSZศརͦ͏ – ("ͷੜϩάΛ#JH2VFSZͰղੳͰ͖ΔΦϓγϣϯ – ͨͩ͠("ͷ༗ྉαʔϏε • Ͱ͔͍σʔλͷΠϯϙʔτ
– (PPHMF%BUB4UPSFʹஔ͍͔ͯΒΠϯϙʔτ͢Δͱߴ • 5BCMF%FDPSBUPST – σʔλͷ࣌ؒൣғΛࢦఆͯ͠ΫΤϦɻεΩϟϯରͷσʔλ͕খ͘͞ͳ ΔͷͰΫΤϦඅ༻ΛઅͰ͖Δ • +0*/੍ݶ.#ੲͷ – +0*/&"$)Λ͏ͱ.BQ3FEVDFͷTIV⒐FΈ͍ͨͳॲཧͰڊ େͳ+0*/ ԯYԯͱ͔ŋŋŋ ͯ͘͠ΕΔΑ
·ͱΊ • #JH2VFSZϑϧεΩϟϯͰͰ͔͍σʔλͷ 42-͕ඵͳαʔϏε • ΫιΫΤϦྗۀͰॲཧͪ͠Ό͏ΧοίΠΠ • ׂ౷࣏ (PPHMFͷ%$نͰ֖ͳ͍ ฒྻॲཧܥ
• όονɺϩάղੳͳΜ͔ʹ͑·͢ • ࢲ(PPHMFࣾͷճ͠ऀͰ͍͟͝·ͤΜ
5IBOLT ֆCZ͋ΘΏ͖