Share
Embed
Slide 1
Slide 1 text
© 2020 Snowflake Inc. All Rights Reserved
Snowflake
July Tech Festa 2021 winter
Yoshi Matsuzaki, Principal Cloud Support Engineer of Snowflake | Dec 2, 2020
Slide 2
Slide 2 text
© 2020 Snowflake Inc. All Rights Reserved
ABOUT ME
2
Yoshi Matsuzaki
Principal Cloud Support Engineer - Snowflake
Work:
Technical Support in English/Japanese (7:3)
Launching Japanese language support team
SQL functionality expert
Escalation review
Supportability readiness for new features
Background:
Infrastructure Engineer (MSP)
Support Engineer (Cloud Vendor)
Database Engineer of MySQL engines (Cloud Vendor)
Slide 3
Slide 3 text
© 2020 Snowflake Inc. All Rights Reserved
SNOWFLAKE?
3
Slide 4
Slide 4 text
© 2020 Snowflake Inc. All Rights Reserved
SNOWFLAKE?
4
(Micro-partitioning)
/ ( )
/ ( )
( × )
90 /
...
Slide 5
Slide 5 text
© 2020 Snowflake Inc. All Rights Reserved
SNOWFLAKE?
5
DATA
SOURCES
OLTP DATABASES
ENTERPRISE
APPLICATIONS
THIRD-PARTY
WEB/LOG DATA
IoT
DATA
CONSUMERS
DATA MONETIZATION
OPERATIONAL
REPORTING
AD HOC ANALYSIS
REAL-TIME ANALYTICS
→ Data Platform → Data Cloud
Slide 6
Slide 6 text
© 2020 Snowflake Inc. All Rights Reserved
DATA CLOUD?
6
"The Data Cloud is a network that connects Snowflake customers,
partners, data providers, and data service providers."
( )
( )
Slide 7
Slide 7 text
© 2020 Snowflake Inc. All Rights Reserved
DATA CLOUD?
7
Zero-Copy Cloning
Data Sharing
Data Exchange
Snowflake Data Marketplace
Slide 8
Slide 8 text
© 2020 Snowflake Inc. All Rights Reserved
DATA CLOUD?
Zero-Copy Cloning
Data Exchange Snowflake Data Marketplace
Data Sharing
Slide 9
Slide 9 text
© 2020 Snowflake Inc. All Rights Reserved
CHARACTERISTICS
9
Slide 10
Slide 10 text
© 2020 Snowflake Inc. All Rights Reserved
ISOLATED STORAGE?
10
Slide 11
Slide 11 text
© 2020 Snowflake Inc. All Rights Reserved
MICRO-PARTITIONING
11
(Micro-Partitions)
Snowflake
/
50-500 MB
( : S3)
… Partition Pruning
/
NDV ( )
Slide 12
Slide 12 text
© 2020 Snowflake Inc. All Rights Reserved 12
Slide 13
Slide 13 text
© 2020 Snowflake Inc. All Rights Reserved
BENEFITS OF MICRO-PARTIONING
13
Slide 14
Slide 14 text
© 2020 Snowflake Inc. All Rights Reserved
ZERO-COPY CLONING
14
'a' 'b' 'c'
Table A Partitions
Table A'
Slide 15
Slide 15 text
© 2020 Snowflake Inc. All Rights Reserved
DATA SHARING
15
'a' 'b' 'c'
Table A Partitions
Account A
Account B Table A
Slide 16
Slide 16 text
© 2020 Snowflake Inc. All Rights Reserved
BENEFITS OF MICRO-PARTIONING
16
…
Slide 17
Slide 17 text
© 2020 Snowflake Inc. All Rights Reserved
BENEFITS OF MICRO-PARTITIONING
17
S3
( )
Slide 18
Slide 18 text
© 2020 Snowflake Inc. All Rights Reserved
BENEFITS OF MICRO-PARTITIONING
18
/
GC
=
(Time Travel)
Slide 19
Slide 19 text
© 2020 Snowflake Inc. All Rights Reserved 19
TIME TRAVEL
'a' 'b' 'c'
Table Partitions
UPDATE tab SET col = 'd'
WHERE col = 'a';
'a' 'b' 'c'
Table Partitions
'd'
v1 v2
Slide 20
Slide 20 text
© 2020 Snowflake Inc. All Rights Reserved 20
TIME TRAVEL
Table
SELECT col FROM tab;
'a' 'b' 'c'
Partitions
'd'
v1 v2
SELECT col FROM tab
BEFORE (STATEMENT = ...);
a
b
c
b
c
d
Slide 21
Slide 21 text
© 2020 Snowflake Inc. All Rights Reserved 21
TIME TRAVEL
Table t_old
'a' 'b' 'c'
Partitions
'd'
v1 v2
CREATE TABLE t_old CLONE t
BEFORE (STATEMENT = ...);
Table t
Slide 22
Slide 22 text
© 2020 Snowflake Inc. All Rights Reserved 22
TIME TRAVEL & FAIL SAFE
0 1 2 90 91 97
89
Time Travel ( / ) Fail Safe ( )
0 90 ( ) Time Travel 7
Slide 23
Slide 23 text
© 2020 Snowflake Inc. All Rights Reserved 23
TIME TRAVEL & FAIL SAFE
0 1 2 90 91 97
89
Time Travel ( / ) Fail Safe ( )
98
0 90 ( ) Time Travel 7
Slide 24
Slide 24 text
© 2020 Snowflake Inc. All Rights Reserved 24
TIME TRAVEL & FAIL SAFE
0 1 2 90 91 97
89
Time Travel ( / ) Fail Safe ( )
98
GC
0 90 ( ) Time Travel 7
Slide 25
Slide 25 text
© 2020 Snowflake Inc. All Rights Reserved
TIME TRAVEL & FAIL SAFE
25
Time Travel
90 /
CREATE TABLE ... CLONE ... ...
SELECT ... FROM ... ...
Fail Safe
Time Travel 7
GC
Slide 26
Slide 26 text
© 2020 Snowflake Inc. All Rights Reserved
BENEFITS OF STORAGE ISOLATION
26
Slide 27
Slide 27 text
© 2020 Snowflake Inc. All Rights Reserved
BENEFITS OF STORAGE ISOLATION
27
(I/O )
( )
( )
Slide 28
Slide 28 text
© 2020 Snowflake Inc. All Rights Reserved
BENEFITS OF STORAGE ISOLATION
28
/
/
/
Slide 29
Slide 29 text
© 2020 Snowflake Inc. All Rights Reserved
BENEFITS OF STORAGE ISOLATION
29
(cont.)
/
( )
…
Slide 30
Slide 30 text
© 2020 Snowflake Inc. All Rights Reserved
SNOWFLAKE ARCHITECTURE
30
Slide 31
Slide 31 text
© 2020 Snowflake Inc. All Rights Reserved
REAL-WORLD USE CASE
31
Continuous
Loading (4TB/day) S3
<5min SLA
Virtual Warehouse
Medium
ETL &
Maintenance
Virtual Warehouse
Large
4 trillion rows
3+ petabyte raw data
8x compression ratio
25M micro partitions
Prod DB
Virtual Warehouse
2X-Large
Reporting
(Segmented)
Interactive
Dashboard
50% < 1s
85% < 2s
95% < 5s
Virtual Warehouse
Auto Scale – X-Large x 5
Slide 32
Slide 32 text
© 2020 Snowflake Inc. All Rights Reserved
ARCHITECTURE TO X
32
→
→
→
(Extensibility)
→
Data Warehouse → Data Platform → Data Cloud → ...
Slide 33
Slide 33 text
© 2020 Snowflake Inc. All Rights Reserved
SNOWFLAKE?
33
Slide 34
Slide 34 text
© 2020 Snowflake Inc. All Rights Reserved