Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Snowflake のアーキテクチャが どう「筋がいい」のかを解説する / Data Engineering Study #5

Snowflake のアーキテクチャが どう「筋がいい」のかを解説する / Data Engineering Study #5

Data Engineering Study #5「噂のSnowflake Deep Dive」発表資料: https://forkwell.connpass.com/event/194269/
Youtube Live アーカイブ: https://www.youtube.com/watch?v=_kYW7EneUu4

11d5350db3f389d0f5749b91ee11a916?s=128

Yoshi Matsuzaki

December 03, 2020
Tweet

Transcript

  1. © 2020 Snowflake Inc. All Rights Reserved Snowflake Data Engineering

    Study #5 Yoshi Matsuzaki, Principal Cloud Support Engineer of Snowflake | Dec 2, 2020
  2. © 2020 Snowflake Inc. All Rights Reserved ABOUT ME 2

    Yoshi Matsuzaki Principal Cloud Support Engineer - Snowflake Work: Technical Support in English/Japanese (7:3) Launching Japanese language support team SQL functionality expert Escalation review Supportability readiness for new features Background: Infrastructure Engineer (MSP) Support Engineer (Cloud Vendor) Database Engineer of MySQL engines (Cloud Vendor)
  3. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 3

  4. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 4 DATA

    SOURCES OLTP DATABASES ENTERPRISE APPLICATIONS THIRD-PARTY WEB/LOG DATA IoT DATA CONSUMERS DATA MONETIZATION OPERATIONAL REPORTING AD HOC ANALYSIS REAL-TIME ANALYTICS → Data Platform → Data Cloud
  5. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE ARCHITECTURE 5

  6. © 2020 Snowflake Inc. All Rights Reserved Snowflake 6

  7. © 2020 Snowflake Inc. All Rights Reserved 7 Snowflake 50-500

    MB ( : S3) … Partition Pruning / NDV ( )
  8. © 2020 Snowflake Inc. All Rights Reserved 8

  9. © 2020 Snowflake Inc. All Rights Reserved 9 ( )

    “ ( : immutable) ” - Wikipedia
  10. © 2020 Snowflake Inc. All Rights Reserved 10 : (UPDATE)

    : → I/O … : →
  11. © 2020 Snowflake Inc. All Rights Reserved 11 'a' 'b'

    'c' Table Partitions UPDATE tab SET col = 'd' WHERE col = 'a'; 'a' 'b' 'c' Table Partitions 'd' v1 v2
  12. © 2020 Snowflake Inc. All Rights Reserved 12 I/O …

    OLAP OLTP OLAP →
  13. © 2020 Snowflake Inc. All Rights Reserved 13 / /

    : UNDO (MySQL, Oracle) UNDO (PostgreSQL) :
  14. © 2020 Snowflake Inc. All Rights Reserved 14 → /

    → Time Travel 90 / SELECT * FROM tab AT (TIMESTAMP => ...); SELECT * FROM tab BEFORE (STATEMENT => <query-id>); CREATE TABLE tab CLONE src AT (TIMESTAMP => ...); → Fail Safe 7 GC ...
  15. © 2020 Snowflake Inc. All Rights Reserved 15 Table SELECT

    col FROM tab; 'a' 'b' 'c' Partitions 'd' v1 v2 SELECT col FROM tab BEFORE (STATEMENT = ...); a b c b c d
  16. © 2020 Snowflake Inc. All Rights Reserved 16 ( )

    : M:N →
  17. © 2020 Snowflake Inc. All Rights Reserved 17 ... /

    Zero Copy Cloning Data Sharing, Data Exchange ...
  18. © 2020 Snowflake Inc. All Rights Reserved 18 Zero Copy

    Cloning → Data Sharing ( ) → Data Exchange (1:1 ) → Snowflake Data Marketplace ( ) → Snowflake Data Service Marketplace ( ) → ...
  19. © 2020 Snowflake Inc. All Rights Reserved 19 → →

    →… (Extensibility)
  20. © 2020 Snowflake Inc. All Rights Reserved 20 Extensibility (

    ) Snowflake Snowflake extensible → (MP → S/C ) → (S/C → Zero Copy Cloning) → (Cloning → Data Sharing → ...) Snowflake
  21. © 2020 Snowflake Inc. All Rights Reserved 21 ... but

    ( ) ... but ...
  22. © 2020 Snowflake Inc. All Rights Reserved 22 → →

    Snowflake →
  23. © 2020 Snowflake Inc. All Rights Reserved Snowflake

  24. 2020 12 10 11:00 12:00am Live Snowflake Snowflake trocco Snowflake

    primeNumber
  25. © 2020 Snowflake Inc. All Rights Reserved