Upgrade to Pro — share decks privately, control downloads, hide ads and more …

「データクラウド」を実現する Snowflake のアーキテクチャ / July Tech Festa 2021 winter

「データクラウド」を実現する Snowflake のアーキテクチャ / July Tech Festa 2021 winter

July Tech Festa 2021 winter 発表資料:
https://techfesta.connpass.com/event/193966/
YouTube: https://youtu.be/k2l2ZHoV3js

Yoshi Matsuzaki

January 24, 2021
Tweet

More Decks by Yoshi Matsuzaki

Other Decks in Technology

Transcript

  1. © 2020 Snowflake Inc. All Rights Reserved Snowflake July Tech

    Festa 2021 winter Yoshi Matsuzaki, Principal Cloud Support Engineer of Snowflake | Dec 2, 2020
  2. © 2020 Snowflake Inc. All Rights Reserved ABOUT ME 2

    Yoshi Matsuzaki Principal Cloud Support Engineer - Snowflake Work: Technical Support in English/Japanese (7:3) Launching Japanese language support team SQL functionality expert Escalation review Supportability readiness for new features Background: Infrastructure Engineer (MSP) Support Engineer (Cloud Vendor) Database Engineer of MySQL engines (Cloud Vendor)
  3. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 5 DATA

    SOURCES OLTP DATABASES ENTERPRISE APPLICATIONS THIRD-PARTY WEB/LOG DATA IoT DATA CONSUMERS DATA MONETIZATION OPERATIONAL REPORTING AD HOC ANALYSIS REAL-TIME ANALYTICS → Data Platform → Data Cloud
  4. © 2020 Snowflake Inc. All Rights Reserved DATA CLOUD? 6

    "The Data Cloud is a network that connects Snowflake customers, partners, data providers, and data service providers." ( ) ( )
  5. © 2020 Snowflake Inc. All Rights Reserved DATA CLOUD? 7

    Zero-Copy Cloning Data Sharing Data Exchange Snowflake Data Marketplace
  6. © 2020 Snowflake Inc. All Rights Reserved DATA CLOUD? Zero-Copy

    Cloning Data Exchange Snowflake Data Marketplace Data Sharing
  7. © 2020 Snowflake Inc. All Rights Reserved MICRO-PARTITIONING 11 (Micro-Partitions)

    Snowflake / 50-500 MB ( : S3) … Partition Pruning / NDV ( )
  8. © 2020 Snowflake Inc. All Rights Reserved ZERO-COPY CLONING 14

    'a' 'b' 'c' Table A Partitions Table A'
  9. © 2020 Snowflake Inc. All Rights Reserved DATA SHARING 15

    'a' 'b' 'c' Table A Partitions Account A Account B Table A
  10. © 2020 Snowflake Inc. All Rights Reserved 19 TIME TRAVEL

    'a' 'b' 'c' Table Partitions UPDATE tab SET col = 'd' WHERE col = 'a'; 'a' 'b' 'c' Table Partitions 'd' v1 v2
  11. © 2020 Snowflake Inc. All Rights Reserved 20 TIME TRAVEL

    Table SELECT col FROM tab; 'a' 'b' 'c' Partitions 'd' v1 v2 SELECT col FROM tab BEFORE (STATEMENT = ...); a b c b c d
  12. © 2020 Snowflake Inc. All Rights Reserved 21 TIME TRAVEL

    Table t_old 'a' 'b' 'c' Partitions 'd' v1 v2 CREATE TABLE t_old CLONE t BEFORE (STATEMENT = ...); Table t
  13. © 2020 Snowflake Inc. All Rights Reserved 22 TIME TRAVEL

    & FAIL SAFE 0 1 2 90 91 97 89 Time Travel ( / ) Fail Safe ( ) 0 90 ( ) Time Travel 7
  14. © 2020 Snowflake Inc. All Rights Reserved 23 TIME TRAVEL

    & FAIL SAFE 0 1 2 90 91 97 89 Time Travel ( / ) Fail Safe ( ) 98 0 90 ( ) Time Travel 7
  15. © 2020 Snowflake Inc. All Rights Reserved 24 TIME TRAVEL

    & FAIL SAFE 0 1 2 90 91 97 89 Time Travel ( / ) Fail Safe ( ) 98 GC 0 90 ( ) Time Travel 7
  16. © 2020 Snowflake Inc. All Rights Reserved TIME TRAVEL &

    FAIL SAFE 25 Time Travel 90 / CREATE TABLE ... CLONE ... <BEFORE|AT> ... SELECT ... FROM ... <BEFORE|AT> ... Fail Safe Time Travel 7 GC
  17. © 2020 Snowflake Inc. All Rights Reserved REAL-WORLD USE CASE

    31 Continuous Loading (4TB/day) S3 <5min SLA Virtual Warehouse Medium ETL & Maintenance Virtual Warehouse Large 4 trillion rows 3+ petabyte raw data 8x compression ratio 25M micro partitions Prod DB Virtual Warehouse 2X-Large Reporting (Segmented) Interactive Dashboard 50% < 1s 85% < 2s 95% < 5s Virtual Warehouse Auto Scale – X-Large x 5
  18. © 2020 Snowflake Inc. All Rights Reserved ARCHITECTURE TO X

    32 → → → (Extensibility) → Data Warehouse → Data Platform → Data Cloud → ...