「データクラウド」を実現する Snowflake のアーキテクチャ / July Tech Festa 2021 winter

July Tech Festa 2021 winter 発表資料:
https://techfesta.connpass.com/event/193966/
YouTube: https://youtu.be/k2l2ZHoV3js

Transcript

1. © 2020 Snowflake Inc. All Rights Reserved Snowflake July Tech Festa 2021 winter Yoshi Matsuzaki, Principal Cloud Support Engineer of Snowflake | Dec 2, 2020
2. © 2020 Snowflake Inc. All Rights Reserved ABOUT ME 2 Yoshi Matsuzaki Principal Cloud Support Engineer - Snowflake Work: Technical Support in English/Japanese (7:3) Launching Japanese language support team SQL functionality expert Escalation review Supportability readiness for new features Background: Infrastructure Engineer (MSP) Support Engineer (Cloud Vendor) Database Engineer of MySQL engines (Cloud Vendor)
3. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 3
4. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 4 (Micro-partitioning) / ( ) / ( ) ( × ) 90 / ...
5. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 5 DATA SOURCES OLTP DATABASES ENTERPRISE APPLICATIONS THIRD-PARTY WEB/LOG DATA IoT DATA CONSUMERS DATA MONETIZATION OPERATIONAL REPORTING AD HOC ANALYSIS REAL-TIME ANALYTICS → Data Platform → Data Cloud
6. © 2020 Snowflake Inc. All Rights Reserved DATA CLOUD? 6 "The Data Cloud is a network that connects Snowflake customers, partners, data providers, and data service providers." ( ) ( )
7. © 2020 Snowflake Inc. All Rights Reserved DATA CLOUD? 7 Zero-Copy Cloning Data Sharing Data Exchange Snowflake Data Marketplace
8. © 2020 Snowflake Inc. All Rights Reserved DATA CLOUD? Zero-Copy Cloning Data Exchange Snowflake Data Marketplace Data Sharing
9. © 2020 Snowflake Inc. All Rights Reserved CHARACTERISTICS 9
10. © 2020 Snowflake Inc. All Rights Reserved ISOLATED STORAGE? 10
11. © 2020 Snowflake Inc. All Rights Reserved MICRO-PARTITIONING 11 (Micro-Partitions) Snowflake / 50-500 MB ( : S3) … Partition Pruning / NDV ( )
12. © 2020 Snowflake Inc. All Rights Reserved 12
13. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF MICRO-PARTIONING 13
14. © 2020 Snowflake Inc. All Rights Reserved ZERO-COPY CLONING 14 'a' 'b' 'c' Table A Partitions Table A'
15. © 2020 Snowflake Inc. All Rights Reserved DATA SHARING 15 'a' 'b' 'c' Table A Partitions Account A Account B Table A
16. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF MICRO-PARTIONING 16 …
17. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF MICRO-PARTITIONING 17 S3 ( )
18. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF MICRO-PARTITIONING 18 / GC = (Time Travel)
19. © 2020 Snowflake Inc. All Rights Reserved 19 TIME TRAVEL 'a' 'b' 'c' Table Partitions UPDATE tab SET col = 'd' WHERE col = 'a'; 'a' 'b' 'c' Table Partitions 'd' v1 v2
20. © 2020 Snowflake Inc. All Rights Reserved 20 TIME TRAVEL Table SELECT col FROM tab; 'a' 'b' 'c' Partitions 'd' v1 v2 SELECT col FROM tab BEFORE (STATEMENT = ...); a b c b c d
21. © 2020 Snowflake Inc. All Rights Reserved 21 TIME TRAVEL Table t_old 'a' 'b' 'c' Partitions 'd' v1 v2 CREATE TABLE t_old CLONE t BEFORE (STATEMENT = ...); Table t
22. © 2020 Snowflake Inc. All Rights Reserved 22 TIME TRAVEL & FAIL SAFE 0 1 2 90 91 97 89 Time Travel ( / ) Fail Safe ( ) 0 90 ( ) Time Travel 7
23. © 2020 Snowflake Inc. All Rights Reserved 23 TIME TRAVEL & FAIL SAFE 0 1 2 90 91 97 89 Time Travel ( / ) Fail Safe ( ) 98 0 90 ( ) Time Travel 7
24. © 2020 Snowflake Inc. All Rights Reserved 24 TIME TRAVEL & FAIL SAFE 0 1 2 90 91 97 89 Time Travel ( / ) Fail Safe ( ) 98 GC 0 90 ( ) Time Travel 7
25. © 2020 Snowflake Inc. All Rights Reserved TIME TRAVEL & FAIL SAFE 25 Time Travel 90 / CREATE TABLE ... CLONE ... <BEFORE|AT> ... SELECT ... FROM ... <BEFORE|AT> ... Fail Safe Time Travel 7 GC
26. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF STORAGE ISOLATION 26
27. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF STORAGE ISOLATION 27 (I/O ) ( ) ( )
28. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF STORAGE ISOLATION 28 / / /
29. © 2020 Snowflake Inc. All Rights Reserved BENEFITS OF STORAGE ISOLATION 29 (cont.) / ( ) …
30. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE ARCHITECTURE 30
31. © 2020 Snowflake Inc. All Rights Reserved REAL-WORLD USE CASE 31 Continuous Loading (4TB/day) S3 <5min SLA Virtual Warehouse Medium ETL & Maintenance Virtual Warehouse Large 4 trillion rows 3+ petabyte raw data 8x compression ratio 25M micro partitions Prod DB Virtual Warehouse 2X-Large Reporting (Segmented) Interactive Dashboard 50% < 1s 85% < 2s 95% < 5s Virtual Warehouse Auto Scale – X-Large x 5
32. © 2020 Snowflake Inc. All Rights Reserved ARCHITECTURE TO X 32 → → → (Extensibility) → Data Warehouse → Data Platform → Data Cloud → ...
33. © 2020 Snowflake Inc. All Rights Reserved SNOWFLAKE? 33
34. © 2020 Snowflake Inc. All Rights Reserved