Upgrade to Pro — share decks privately, control downloads, hide ads and more …

「データクラウド」を実現する Snowflake のアーキテクチャ / July Tech Festa 2021 winter

「データクラウド」を実現する Snowflake のアーキテクチャ / July Tech Festa 2021 winter

July Tech Festa 2021 winter 発表資料:
https://techfesta.connpass.com/event/193966/
YouTube: https://youtu.be/k2l2ZHoV3js

11d5350db3f389d0f5749b91ee11a916?s=128

Yoshi Matsuzaki

January 24, 2021
Tweet

Transcript

  1. © 2020 Snowflake Inc. All Rights Reserved Snowflake July Tech

    Festa 2021 winter Yoshi Matsuzaki, Principal Cloud Support Engineer of Snowflake | Dec 2, 2020
  2. © 2020 Snowflake Inc. All Rights Reserved ABOUT ME 2

    Yoshi Matsuzaki Principal Cloud Support Engineer - Snowflake Work: Technical Support in English/Japanese (7:3) Launching Japanese language support team SQL functionality expert Escalation review Supportability readiness for new features Background: Infrastructure Engineer (MSP) Support Engineer (Cloud Vendor) Database Engineer of MySQL engines (Cloud Vendor)
  3. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 3

  4. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 4 (Micro-partitioning)

    / ( ) / ( ) ( × ) 90 / ...
  5. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 5 DATA

    SOURCES OLTP DATABASES ENTERPRISE APPLICATIONS THIRD-PARTY WEB/LOG DATA IoT DATA CONSUMERS DATA MONETIZATION OPERATIONAL REPORTING AD HOC ANALYSIS REAL-TIME ANALYTICS → Data Platform → Data Cloud
  6. © 2020 Snowflake Inc. All Rights Reserved DATA CLOUD? 6

    "The Data Cloud is a network that connects Snowflake customers, partners, data providers, and data service providers." ( ) ( )
  7. © 2020 Snowflake Inc. All Rights Reserved DATA CLOUD? 7

    Zero-Copy Cloning Data Sharing Data Exchange Snowflake Data Marketplace
  8. © 2020 Snowflake Inc. All Rights Reserved DATA CLOUD? Zero-Copy

    Cloning Data Exchange Snowflake Data Marketplace Data Sharing
  9. © 2020 Snowflake Inc. All Rights Reserved CHARACTERISTICS 9

  10. © 2020 Snowflake Inc. All Rights Reserved ISOLATED STORAGE? 10

  11. © 2020 Snowflake Inc. All Rights Reserved MICRO-PARTITIONING 11 (Micro-Partitions)

    Snowflake / 50-500 MB ( : S3) … Partition Pruning / NDV ( )
  12. © 2020 Snowflake Inc. All Rights Reserved 12

  13. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF MICRO-PARTIONING

    13
  14. © 2020 Snowflake Inc. All Rights Reserved ZERO-COPY CLONING 14

    'a' 'b' 'c' Table A Partitions Table A'
  15. © 2020 Snowflake Inc. All Rights Reserved DATA SHARING 15

    'a' 'b' 'c' Table A Partitions Account A Account B Table A
  16. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF MICRO-PARTIONING

    16 …
  17. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF MICRO-PARTITIONING

    17 S3 ( )
  18. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF MICRO-PARTITIONING

    18 / GC = (Time Travel)
  19. © 2020 Snowflake Inc. All Rights Reserved 19 TIME TRAVEL

    'a' 'b' 'c' Table Partitions UPDATE tab SET col = 'd' WHERE col = 'a'; 'a' 'b' 'c' Table Partitions 'd' v1 v2
  20. © 2020 Snowflake Inc. All Rights Reserved 20 TIME TRAVEL

    Table SELECT col FROM tab; 'a' 'b' 'c' Partitions 'd' v1 v2 SELECT col FROM tab BEFORE (STATEMENT = ...); a b c b c d
  21. © 2020 Snowflake Inc. All Rights Reserved 21 TIME TRAVEL

    Table t_old 'a' 'b' 'c' Partitions 'd' v1 v2 CREATE TABLE t_old CLONE t BEFORE (STATEMENT = ...); Table t
  22. © 2020 Snowflake Inc. All Rights Reserved 22 TIME TRAVEL

    & FAIL SAFE 0 1 2 90 91 97 89 Time Travel ( / ) Fail Safe ( ) 0 90 ( ) Time Travel 7
  23. © 2020 Snowflake Inc. All Rights Reserved 23 TIME TRAVEL

    & FAIL SAFE 0 1 2 90 91 97 89 Time Travel ( / ) Fail Safe ( ) 98 0 90 ( ) Time Travel 7
  24. © 2020 Snowflake Inc. All Rights Reserved 24 TIME TRAVEL

    & FAIL SAFE 0 1 2 90 91 97 89 Time Travel ( / ) Fail Safe ( ) 98 GC 0 90 ( ) Time Travel 7
  25. © 2020 Snowflake Inc. All Rights Reserved TIME TRAVEL &

    FAIL SAFE 25 Time Travel 90 / CREATE TABLE ... CLONE ... <BEFORE|AT> ... SELECT ... FROM ... <BEFORE|AT> ... Fail Safe Time Travel 7 GC
  26. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF STORAGE

    ISOLATION 26
  27. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF STORAGE

    ISOLATION 27 (I/O ) ( ) ( )
  28. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF STORAGE

    ISOLATION 28 / / /
  29. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF STORAGE

    ISOLATION 29 (cont.) / ( ) …
  30. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE ARCHITECTURE 30

  31. © 2020 Snowflake Inc. All Rights Reserved REAL-WORLD USE CASE

    31 Continuous Loading (4TB/day) S3 <5min SLA Virtual Warehouse Medium ETL & Maintenance Virtual Warehouse Large 4 trillion rows 3+ petabyte raw data 8x compression ratio 25M micro partitions Prod DB Virtual Warehouse 2X-Large Reporting (Segmented) Interactive Dashboard 50% < 1s 85% < 2s 95% < 5s Virtual Warehouse Auto Scale – X-Large x 5
  32. © 2020 Snowflake Inc. All Rights Reserved ARCHITECTURE TO X

    32 → → → (Extensibility) → Data Warehouse → Data Platform → Data Cloud → ...
  33. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 33

  34. © 2020 Snowflake Inc. All Rights Reserved