Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Google BigQuery の話 #yapcasia

Naoya Ito
August 30, 2014

Google BigQuery の話 #yapcasia

Naoya Ito

August 30, 2014
Tweet

More Decks by Naoya Ito

Other Decks in Technology

Transcript

  1. #JH2VFSZͱ͸ •  ڊେͳσʔλ΁ͷ42- ͳͲ Λ਺ඵͰ࣮ߦ͢ΔΫϥ΢υαʔϏε –  ԯϨίʔυΛඵ ˞  – 

    8FCΠϯλʔϑΣʔε͓Αͼ3&45"1* •  (PPHMFࣾ಺Ͱ࢖ΘΕ͖ͯͨ%SFNFMΛαʔϏεԽ –  ೥݄$MPTFEϦϦʔε –  ೥݄Ұൠެ։ –  ܧଓతʹόʔδϣϯΞοϓ –  ೥݄#JH2VFSZ4USFBNJOH ˞(PPHMFͷދͷࢠʮ#JH2VFSZʯΛ'MVFOUEϢʔβʔ͕࢖Θͳ͍ཧ༝͕ͳ͘ͳͬͨཧ༝ IUUQRJJUBDPNLB[VOPSJJUFNTBDBDCCBBBG
  2. #JH2VFSZ͸ͳͥ଎͍͔ •  جຊɺϑϧεΩϟϯͰ͕Μ͹Δ –  3%#.4ͷ#5SFFΠϯσοΫεͱ͔ͳ͍ •  42-Λ෼ࢄॲཧ –  .11 .BTTJWFMZ1BSBMMFM1SPDFTTJOH

    2VFSZ&OHJOF %SFNFM •  ਺ઍ୆ͷσΟεΫͱߴ଎ωοτϫʔΫͰεέʔϧΞ΢τ –  5#ͷσʔλΛඵͰϦʔυ͢Δ*0
  3. ଞͷྨࣅ࣮૷ͱͷϙδγϣχϯά •  -BSHF#BUDI –  ҆ఆͯ͠ڊେͳόονΛ࣮ߦͰ͖Δ –  ΫΤϦ࣮ߦ࣌ͷΦʔόʔϔου͕େ͖͍ ਺ेඵʙ਺े෼  – 

    .BQ3FEVDFɺ)BEPPQ )JWF  •  4IPSU#BUDI –  ΫΤϦ࣮ߦ࣌ͷΦʔόʔϔου͕਺ඵ –  ΞυϗοΫΫΤϦʹ޲͍͍ͯΔ –  .112VFSZ&OHJOF1SFTUPɺ*NQBMBɺ#JH2VFSZ %SFNFM   •  4USFBN1SPDFTTJOH –  όον͸࣮ߦͰ͖ͳ͍͕ετϦʔϜʹରͯ͠ϦΞϧλΠϜॲཧͰ͖Δ –  /PSJLSBɺ"QBDIF,BGLBɺ5XJUUFS4UPSNFUD "NB[PO3FETIJGU΋ 4IPSU#BUDI  ৄ͘͠ ͳ͍ͷͰলུ  cf.  Batch  processing  and  Stream  processing  by  SQL   h;p://www.slideshare.net/tagomoris/hcj2014-­‐sql
  4. (PPHMF#JH%BUB4UBDL •  ʰ(PPHMFΛࢧ͑Δٕज़ʱ –  #JH%BUB4UBDL –  ('4ɺ#JH5BCMFɺ.BQ3FEVDFFUD •  #JH%BUB4UBDL – 

    #JH%BUB4UBDLͷ্ʹߏங͞Εͨɺͷ՝୊Λղফ͢Δ࣮૷܈ –  $PMPTTVT .FHBTUPSF 4QBOOFS 'MVNF+BWB %SFNFM طʹ(PPHMFࣾ಺͸ #JH%BUB4UBDLͩ ͱ͔͍͏࿩΋ͪΒ΄Β
  5. #JH2VFSZͷٕज़ελοΫ (PPHMF'JMF4ZTUFN ('4  $PMPTTVT'JMF4ZTUFN $'4  $PMVNO*0 %SFNFM ෼ࢄ'4

    ('4ͷվྑܕ'4 ৄࡉ͸ඇެ։  #JH2VFSZͷͨΊͷྻࢦ޲ϑΝΠϧ ϑΥʔϚοτ ฒྻ42-࣮ߦΤϯδϯ σʔληϯλʔΛ·͍ͨͰ ෼ࢄ͞ΕͯΔσʔλΛฒྻ ͔ͭߴ଎ʹऔಘͰ͖ΔΒ͠ ͍
  6. Root  Mixer Mixer  1   Shard  0-­‐8 Mixer  1  

    Shard  9-­‐16 Mixer  1   Shard  17-­‐24 Shard  0 Shard  10 Shard  12 Shard  20 Shard  24 Distributed  Storage  (e.g.,  CFS) Dremel  serving  tree Google  BigQuery  AnalyIcs  P.284   Chapter  9  Understanding  Query  ExecuIon ෼ࢄ
  7. Root  Mixer Mixer  1   Shard  0-­‐8 Mixer  1  

    Shard  9-­‐16 Mixer  1   Shard  17-­‐24 Shard  0 Shard  10 Shard  12 Shard  20 Shard  24 Distributed  Storage  (e.g.,  CFS) Dremel  serving  tree Google  BigQuery  AnalyIcs  P.284   Chapter  9  Understanding  Query  ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ෼ࢄ ू໿
  8. Root  Mixer Mixer  1   Shard  0-­‐8 Mixer  1  

    Shard  9-­‐16 Mixer  1   Shard  17-­‐24 Shard  0 Shard  10 Shard  12 Shard  20 Shard  24 Distributed  Storage  (e.g.,  CFS) Dremel  serving  tree Google  BigQuery  AnalyIcs  P.284   Chapter  9  Understanding  Query  ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ྻΛॱ൪ʹಡΈߦ Λऔಘɻ8)&3&۟ͳ ͲΛݟͯඞཁͳߦͷΈ ʹߜΓϝϞϦͰอ࣋ ෼ࢄ ू໿
  9. Root  Mixer Mixer  1   Shard  0-­‐8 Mixer  1  

    Shard  9-­‐16 Mixer  1   Shard  17-­‐24 Shard  0 Shard  10 Shard  12 Shard  20 Shard  24 Distributed  Storage  (e.g.,  CFS) Dremel  serving  tree Google  BigQuery  AnalyIcs  P.284   Chapter  9  Understanding  Query  ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ྻΛॱ൪ʹಡΈߦ Λऔಘɻ8)&3&۟ͳ ͲΛݟͯඞཁͳߦͷΈ ʹߜΓϝϞϦͰอ࣋ ֤TIBSE͔ΒσʔλΛू ໿ɻྫ͑͹ιʔτ΍-*.*5 ͷߜΓࠐΈͳͲ͢Δ ෼ࢄ ू໿
  10. Root  Mixer Mixer  1   Shard  0-­‐8 Mixer  1  

    Shard  9-­‐16 Mixer  1   Shard  17-­‐24 Shard  0 Shard  10 Shard  12 Shard  20 Shard  24 Distributed  Storage  (e.g.,  CFS) Dremel  serving  tree Google  BigQuery  AnalyIcs  P.284   Chapter  9  Understanding  Query  ExecuIon $'4 $PMVNO*0Ͱಛ ఆྻͷσʔλ͕Ұ෦ฦͬ ͯ͘Δ ྻΛॱ൪ʹಡΈߦ Λऔಘɻ8)&3&۟ͳ ͲΛݟͯඞཁͳߦͷΈ ʹߜΓϝϞϦͰอ࣋ ֤TIBSE͔ΒσʔλΛू ໿ɻྫ͑͹ιʔτ΍-*.*5 ͷߜΓࠐΈͳͲ͢Δ ू໿ͨ݁͠Ռ ΛDBMMFSʹฦ͢ ෼ࢄ ू໿
  11. ϩάΛඈ͹͢ܖػ •  ϦΫΤετຖ –  "QQMJDBUJPO$POUSPMMFS –  ϩάΠϯϢʔβͷଐੑΛඈ͹͢ˠ%"6΍."6ͷ ࢉग़ʹ •  Ϟσϧͷঢ়ଶมߋ࣌

    –  "DUJWF3FDPSE0CTFSWFS –  Ϟσϧຖʹద౰ͳଐੑΛݟસͬͯඈ͹͢ –  #JH2VFSZ͸ෳࡶͳ42-Ͱ΋ී௨ʹԠ౴͢Δ㱺ϓ ϩμΫτϚωʔδϟ͕ؾܰʹ42-ॻ͍ͯΔ
  12. ֎෦πʔϧͱͷ઀ଓ •  ΤΫηϧ –  #JH2VFSZ$POOFDUPSGPS&YDFMCZ(PPHMF –  ϐϘοτ෼ੳʹ •  %0.0 #*

     –  FYQFSJNFOUBMͳ#JH2VFSZΠϯλϑΣʔε ͋ͬͨ –  5BCMFBV౳ϝδϟʔͲ͜Ζ΋ରԠ࢝͠ΊͯΔ
  13. ໘౗ͳͱ͜Ζ •  qVFOUEQMVHJOCJHRVFSZ͕εΩʔϚϑΝΠϧΛཁٻ ͢Δ –  ͕͔ͩ͠͠IBLPCFSB͞Μ͕QBUDIΛॻ͍ͯ͘Εͨ –  W͔ΒGFUDI@TDIFNBػೳ͕࢖͑ΔΑ •  ࣍ݩςʔϒϧͷߋ৽

    –  61%"5&Ͱ͖ͳ͍ͷͰ –  ໷ؒͱ͔ʹҰճফͯ͠࡞ΔɺΈ͍ͨͳ –  1SFTUPΈ͍ͨʹҧ͏σʔλιʔεΛ+0*/Ͱ͖ͨΓ͢Δͱخ ͍͠ͷ͕ͩŋŋŋ
  14. ࢛ํࢁ •  (PPHMF"OBMZUJDT #JH2VFSZศརͦ͏ –  ("ͷੜϩάΛ#JH2VFSZͰղੳͰ͖ΔΦϓγϣϯ –  ͨͩ͠("ͷ༗ྉαʔϏε •  Ͱ͔͍σʔλͷΠϯϙʔτ

    –  (PPHMF%BUB4UPSFʹஔ͍͔ͯΒΠϯϙʔτ͢Δͱߴ଎ •  5BCMF%FDPSBUPST –  σʔλͷ࣌ؒൣғΛࢦఆͯ͠ΫΤϦɻεΩϟϯର৅ͷσʔλ͕খ͘͞ͳ ΔͷͰΫΤϦඅ༻Λઅ໿Ͱ͖Δ •  +0*/੍ݶ.#͸ੲͷ࿩ –  +0*/&"$)Λ࢖͏ͱ.BQ3FEVDFͷTIV⒐FΈ͍ͨͳॲཧͰڊ େͳ+0*/ ԯYԯͱ͔ŋŋŋ ͯ͘͠ΕΔΑ