Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Google BigQuery の話 #gcpja
Search
Naoya Ito
September 17, 2014
Technology
17
5.4k
Google BigQuery の話 #gcpja
gcp ja night で話した BigQuery のスライド。YAPC::Asia のものに数枚だけスライドを追加したもので、ほぼ同じです。
Naoya Ito
September 17, 2014
Tweet
Share
More Decks by Naoya Ito
See All by Naoya Ito
Functional TypeScript
naoya
13
5.6k
TypeScript 関数型スタイルでバックエンド開発のリアル
naoya
63
28k
シェルの履歴とイクンリメンタル検索を使う
naoya
7
3k
20230227-engineer-type-talk.pdf
naoya
85
40k
関数型プログラミングと型システムのメンタルモデル
naoya
61
89k
TypeScript による GraphQL バックエンド開発
naoya
28
30k
フロントエンドのパラダイムを参考にバックエンド開発を再考する / TypeScript による GraphQL バックエンド開発
naoya
66
23k
「問題から目を背けず取り組む」 一休の開発チームが6年間で学んだこと
naoya
144
58k
一休の現在と、ここまでの道のり
naoya
90
41k
Other Decks in Technology
See All in Technology
.NET 9 のパフォーマンス改善
nenonaninu
0
1.6k
AWS re:Invent 2024 ふりかえり勉強会
yhana
0
590
スケールし続ける事業とサービスを支える組織とアーキテクチャの生き残り戦略 / The survival strategy for Money Forward’s engineering.
moneyforward
0
110
社内イベント管理システムを1週間でAKSからACAに移行した話し
shingo_kawahara
0
210
[トレノケ雲の会 mod.13] 3回目のre:Inventで気づいたこと -CloudOperationsを添えて-
shintaro_fukatsu
0
110
事業貢献を考えるための技術改善の目標設計と改善実績 / Targeted design of technical improvements to consider business contribution and improvement performance
oomatomo
0
170
多領域インシデントマネジメントへの挑戦:ハードウェアとソフトウェアの融合が生む課題/Challenge to multidisciplinary incident management: Issues created by the fusion of hardware and software
bitkey
PRO
2
120
Yahoo! ズバトクにおけるフロントエンド開発
lycorptech_jp
PRO
0
110
3年でバックエンドエンジニアが5倍に増えても破綻しなかったアーキテクチャ そして、これから / Software architecture that scales even with a 5x increase in backend engineers in 3 years
euglena1215
11
4k
LINE Developersプロダクト(LIFF/LINE Login)におけるフロントエンド開発
lycorptech_jp
PRO
0
150
MasterMemory v3 最速確認会
yucchiy
0
200
TSKaigi 2024 の登壇から広がったコミュニティ活動について
tsukuha
0
170
Featured
See All Featured
Facilitating Awesome Meetings
lara
50
6.1k
Art, The Web, and Tiny UX
lynnandtonic
298
20k
Bootstrapping a Software Product
garrettdimon
PRO
305
110k
It's Worth the Effort
3n
183
28k
Imperfection Machines: The Place of Print at Facebook
scottboms
266
13k
What's in a price? How to price your products and services
michaelherold
244
12k
Making Projects Easy
brettharned
116
6k
Rails Girls Zürich Keynote
gr2m
94
13k
Site-Speed That Sticks
csswizardry
2
190
4 Signs Your Business is Dying
shpigford
182
21k
Adopting Sorbet at Scale
ufuk
74
9.1k
The Cult of Friendly URLs
andyhume
78
6.1k
Transcript
(PPHMF#JH2VFSZͷ /BPZB*UP ,"*;&/QMBUGPSN*OD HDQKBOJHIU
ΞδΣϯμ • #JH2VFSZ֓؍ • #JH2VFSZͷ෦ • ,"*;&/QMBUGPSN*ODͰͷ͍Ͳ͜Ζ
#JH2VFSZ֓؍
(PPHMF#JH2VFSZ
None
#JH2VFSZͱ • ڊେͳσʔλͷ42- ͳͲ ΛඵͰ࣮ߦ͢ΔΫϥυαʔϏε – ԯϨίʔυΛඵ ˞ –
8FCΠϯλʔϑΣʔε͓Αͼ3&45"1* • (PPHMFࣾͰΘΕ͖ͯͨ%SFNFMΛαʔϏεԽ – ݄$MPTFEϦϦʔε – ݄Ұൠެ։ – ܧଓతʹόʔδϣϯΞοϓ – ݄#JH2VFSZ4USFBNJOH ˞(PPHMFͷދͷࢠʮ#JH2VFSZʯΛ'MVFOUEϢʔβʔ͕Θͳ͍ཧ༝͕ͳ͘ͳͬͨཧ༝ IUUQRJJUBDPNLB[VOPSJJUFNTBDBDCCBBBG
ͲΜͳ͜ͱʹΘΕΔ͔ • Ϣʔεέʔε – ϩάղੳ – %BUBXBSF)PVTF – • ͍ͯͳ͍༻్ – ۀ%# ͍3%#.4Ͱ
ͳ͍Αɺͱ͍͏͜ͱ
#JH2VFSZͳ͍͔ͥ • جຊɺϑϧεΩϟϯͰ͕ΜΔ – 3%#.4ͷ#5SFFΠϯσοΫεͱ͔ͳ͍ • 42-Λࢄॲཧ – .11 .BTTJWFMZ1BSBMMFM1SPDFTTJOH
2VFSZ&OHJOF %SFNFM • ઍͷσΟεΫͱߴωοτϫʔΫͰεέʔϧΞτ – 5#ͷσʔλΛඵͰϦʔυ͢Δ*0
ͨͩ͠ • ͍3%#.4Ͱͳ͍ • େਓͰҰʹ͏ͷͰͳ͍ – ओʹόονॲཧʹ͏ • εΩʔϚϨεͰͳ͍ 5#نσʔλͰઢܗҎ ԼͰεέʔϧ͢Δ͕ɺٯ
ʹখ͞ͳσʔλͰඵ ͷΦʔόʔϔου͕͋Δ ͷͰ
BigQuery読書会、@harukasan 資料より引用
ଞͷྨࣅ࣮ͱͷϙδγϣχϯά • -BSHF#BUDI – ҆ఆͯ͠ڊେͳόονΛ࣮ߦͰ͖Δ – ΫΤϦ࣮ߦ࣌ͷΦʔόʔϔου͕େ͖͍ ेඵʙे –
.BQ3FEVDFɺ)BEPPQ )JWF • 4IPSU#BUDI – ΫΤϦ࣮ߦ࣌ͷΦʔόʔϔου͕NTʙඵ – ΞυϗοΫΫΤϦʹ͍͍ͯΔ – .112VFSZ&OHJOF1SFTUPɺ*NQBMBɺ#JH2VFSZ %SFNFM • 4USFBN1SPDFTTJOH – όον࣮ߦͰ͖ͳ͍͕ετϦʔϜʹରͯ͠ϦΞϧλΠϜॲཧͰ͖Δ – /PSJLSBɺ"QBDIF,BGLBɺ5XJUUFS4UPSNFUD "NB[PO3FETIJGU 4IPSU#BUDI ৄ͘͠ ͳ͍ͷͰলུ cf. Batch processing and Stream processing by SQL h;p://www.slideshare.net/tagomoris/hcj2014-‐sql
Ձ֨ • ྉۚ – σʔλอ(#݄ – ΫΤϦ5# εΩϟϯͨ͠σʔλͷαΠ ζ "NB[PO4ΑΓ࣮
͍҆ νέοτΒ͍·ͨ͠
#JH2VFSZͷ෦ ͚ͩ͢͜͠
(PPHMF#JH%BUB4UBDL • ʰ(PPHMFΛࢧ͑Δٕज़ʱ – #JH%BUB4UBDL – ('4ɺ#JH5BCMFɺ.BQ3FEVDFFUD • #JH%BUB4UBDL –
#JH%BUB4UBDLͷ্ʹߏங͞Εͨɺͷ՝Λղফ͢Δ࣮܈ – $PMPTTVT .FHBTUPSF 4QBOOFS 'MVNF+BWB %SFNFM طʹ(PPHMFࣾ #JH%BUB4UBDLͩ ͱ͔͍͏ͪΒ΄Β
#JH2VFSZͷٕज़ελοΫ (PPHMF'JMF4ZTUFN ('4 $PMPTTVT'JMF4ZTUFN $'4 $PMVNO*0 %SFNFM ࢄ'4
('4ͷվྑܕ'4 ৄࡉඇެ։ #JH2VFSZͷͨΊͷྻࢦϑΝΠϧ ϑΥʔϚοτ ฒྻ42-࣮ߦΤϯδϯ σʔληϯλʔΛ·͍ͨͰ ࢄ͞ΕͯΔσʔλΛฒྻ ͔ͭߴʹऔಘͰ͖ΔΒ͠ ͍
$PMVNO*0 Dremel: InteracIve Analysis of Web-‐Scale Datasets h;p://research.google.com/pubs/archive/36632.pdf ߦͰͳ͘ྻ୯ҐͰɻಛ
ఆྻΛγʔέϯγϟϧʹ ಡΊΔͭ$PMPTTVT ͰฒྻಡΈࠐΈ
%SFNFM Dremel: InteracIve Analysis of Web-‐Scale Datasets h;p://research.google.com/pubs/archive/36632.pdf
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon ࢄ
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ࢄ ू
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ྻΛॱ൪ʹಡΈߦ Λऔಘɻ8)&3&۟ͳ ͲΛݟͯඞཁͳߦͷΈ ʹߜΓϝϞϦͰอ࣋ ࢄ ू
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ྻΛॱ൪ʹಡΈߦ Λऔಘɻ8)&3&۟ͳ ͲΛݟͯඞཁͳߦͷΈ ʹߜΓϝϞϦͰอ࣋ ֤TIBSE͔ΒσʔλΛू ɻྫ͑ιʔτ-*.*5 ͷߜΓࠐΈͳͲ͢Δ ࢄ ू
Root Mixer Mixer 1 Shard 0-‐8 Mixer 1
Shard 9-‐16 Mixer 1 Shard 17-‐24 Shard 0 Shard 10 Shard 12 Shard 20 Shard 24 Distributed Storage (e.g., CFS) Dremel serving tree Google BigQuery AnalyIcs P.284 Chapter 9 Understanding Query ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ྻΛॱ൪ʹಡΈߦ Λऔಘɻ8)&3&۟ͳ ͲΛݟͯඞཁͳߦͷΈ ʹߜΓϝϞϦͰอ࣋ ֤TIBSE͔ΒσʔλΛू ɻྫ͑ιʔτ-*.*5 ͷߜΓࠐΈͳͲ͢Δ ूͨ݁͠Ռ ΛDBMMFSʹฦ͢ ࢄ ू
#JH2VFSZͷ͍͢͝ॴ • ΧϥϜܕ*0ɺ42-ͷׂ౷࣏ – Ͱ͜Εɺ.11తʹ͘͠ͳ͍ • ͡Ό͋ɺ#JH2VFSZͷԿ͕͍͔͢͝ – (PPHMFͷͰ͔͍Πϯϑϥ
ׂͱ֖ͳ͍ŋŋŋ
͜ΜͳΫιΫΤϦͰඵɺ̐ඵͩ
,"*;&/QMBUGPSN*OD Ͱͷ͍Ͳ͜Ζ
Ϣʔεέʔε • ΞΫηεϩάͷอଘௐࠪ • ΞϓϦέʔγϣϯϩάͷղੳ %BUBXBSF )PVTF • "#ςετͷ༗ҙࠩఆ
ΞΫηεϩά
ΞΫηεϩά #JH2VFSZ • /HJOYͷϩάΛqVFOUQMVHJOCJHRVFSZͰ ૹΓଓ͚Δ – &&Ͱ҉߸Խ͞ΕͯΔΑ • Կ͔༻͕͋ͬͨΒ42-Ͱղੳ –
%BJMZ8FFLMZ.POUIMZ17 – ϓϩμΫγϣϯͷσόοά
qVFOUQMVHJOCJHRVFSZ • CZUBHPNPSJT͞ΜɺZVHVJ͞Μଞ • ઌ͔Β,"*;&/QMBUGPSN*OD͕ϝ ϯςφʹ – ࣮࣭ɺԶ QBUDIFTXFMDPNF Ͱ͢
ΞϓϦέʔγϣϯͷϩάղੳ
ϩάΛඈ͢ • 3BJMT͔ΒUEMPHHFSSVCZͰqVFOUE • qVFOUEQMVHJOCJHRVFSZͰ#2ʹඈ͢
ϩάΛඈ͢ܖػ • ϦΫΤετຖ – "QQMJDBUJPO$POUSPMMFS – ϩάΠϯϢʔβͷଐੑΛඈ͢ˠ%"6."6ͷ ࢉग़ʹ • Ϟσϧͷঢ়ଶมߋ࣌
– "DUJWF3FDPSE0CTFSWFS – ϞσϧຖʹదͳଐੑΛݟસͬͯඈ͢ – #JH2VFSZෳࡶͳ42-Ͱී௨ʹԠ͢Δ㱺ϓ ϩμΫτϚωʔδϟ͕ؾܰʹ42-ॻ͍ͯΔ
ਖ਼نԽ͋·Γ͠ͳ͍ • ελʔεΩʔϚ – %8)ͷఆ൪ͷϞσϦϯά • ϑΝΫτςʔϒϧŋŋŋϩά • ࣍ݩςʔϒϧŋŋŋϚελʔσʔλ ސ٬໊ͱ͔
– ਖ਼نԽ͠ͳ͍ͷ͕ηΦϦʔ
"#ςετ༗ҙࠩఆ • "#ςετͷαʔϏεͳͷͰ͆ • ৄࡉൿີ • SFRTFDͱ͔qVUFOEͰૹͬͯΔ ͚ͲͬͪΌΒ͞ – ˞SFRTFDͷ)551SFRVFTUqVFOUE͕όοϑΝϦϯά͢ΔͷͰ
#JH2VFSZͷ"1*ίʔϧͣͬͱগͳ͍
֎෦πʔϧͱͷଓ • ΤΫηϧ – #JH2VFSZ$POOFDUPSGPS&YDFMCZ(PPHMF – ϐϘοτੳʹ • %0.0 #*
– FYQFSJNFOUBMͳ#JH2VFSZΠϯλϑΣʔε ͋ͬͨ – 5BCMFBVϝδϟʔͲ͜ΖରԠ࢝͠ΊͯΔ
໘ͳͱ͜Ζ • qVFOUEQMVHJOCJHRVFSZ͕εΩʔϚϑΝΠϧΛཁٻ ͢Δ – ͕͔ͩ͠͠IBLPCFSB͞Μ͕QBUDIΛॻ͍ͯ͘Εͨ – W͔ΒGFUDI@TDIFNBػೳ͕͑ΔΑ • ࣍ݩςʔϒϧͷߋ৽
– 61%"5&Ͱ͖ͳ͍ͷͰ – ؒͱ͔ʹҰճফͯ͠࡞ΔɺΈ͍ͨͳ – 1SFTUPΈ͍ͨʹҧ͏σʔλιʔεΛ+0*/Ͱ͖ͨΓ͢Δͱخ ͍͠ͷ͕ͩŋŋŋ
࢛ํࢁͦͷ • 42-ͱ͍ͬͯඪ४42-͡Όͳ͍Α – 3&(&91@."5$) ͱ͔3&(&91@&953"$5 ͱ͔+40/ ͱ ͔501 ͱ͔
• ʮͲ͏ͤϑϧεΩϟϯͯ͠Δ͠ʯͱ͍͏લఏʹཱͭͱΑ ͍ – -&'5 '03."5@65$@64&$ UJNF BTEBZ (3061#:EBZͱ͔ – 3&(&91@&953"$5 UJUMF S aX BTGSBHNFOU(3061#: GSBHNFOU03%&3#:GSBHNFOU@DPVOUEFTDͱ͔ – αϒΫΤϦ7JFX
࢛ํࢁͦͷ • 61%"5&%&-&5&ͳ͍ – ཁΒͳ͍ΧϥϜʹOVMM • ΧϥϜܕ͔ͩΒOVMMͳΒ༰ྔ৯Θͳ͍ – εΩʔϚՃ؆୯ • ߋ৽جຊআͯ͠࡞Γ͠
࢛ํࢁͦͷ • (PPHMF"OBMZUJDT #JH2VFSZศརͦ͏ – ("ͷੜϩάΛ#JH2VFSZͰղੳͰ͖ΔΦϓγϣϯ – ͨͩ͠("ͷ༗ྉαʔϏε • Ͱ͔͍σʔλͷΠϯϙʔτ
– (PPHMF%BUB4UPSFʹஔ͍͔ͯΒΠϯϙʔτ͢Δͱߴ • 5BCMF%FDPSBUPST – σʔλͷ࣌ؒൣғΛࢦఆͯ͠ΫΤϦɻεΩϟϯରͷσʔλ͕খ͘͞ͳ ΔͷͰΫΤϦඅ༻ΛઅͰ͖Δ • +0*/੍ݶ.#ੲͷ – +0*/&"$)Λ͏ͱ.BQ3FEVDFͷTIV⒐FΈ͍ͨͳॲཧͰڊ େͳ+0*/ ԯYԯͱ͔ŋŋŋ ͯ͘͠ΕΔΑ
·ͱΊ • #JH2VFSZϑϧεΩϟϯͰͰ͔͍σʔλͷ 42-͕ඵͳαʔϏε • ΫιΫΤϦྗۀͰॲཧͪ͠Ό͏ΧοίΠΠ • ׂ౷࣏ (PPHMFͷ%$نͰ֖ͳ͍ ฒྻॲཧܥ
• όονɺϩάղੳͳΜ͔ʹ͑·͢ • ࢲ(PPHMFࣾͷճ͠ऀͰ͍͟͝·ͤΜ
5IBOLT ֆCZ͋ΘΏ͖