Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Snowflake のアーキテクチャが どう「筋がいい」のかを解説する / Data Engineering Study #5

Snowflake のアーキテクチャが どう「筋がいい」のかを解説する / Data Engineering Study #5

Data Engineering Study #5「噂のSnowflake Deep Dive」発表資料: https://forkwell.connpass.com/event/194269/
Youtube Live アーカイブ: https://www.youtube.com/watch?v=_kYW7EneUu4

Yoshi Matsuzaki

December 03, 2020
Tweet

More Decks by Yoshi Matsuzaki

Other Decks in Technology

Transcript

  1. © 2020 Snowflake Inc. All Rights Reserved
    Snowflake
    Data Engineering Study #5
    Yoshi Matsuzaki, Principal Cloud Support Engineer of Snowflake | Dec 2, 2020

    View full-size slide

  2. © 2020 Snowflake Inc. All Rights Reserved
    ABOUT ME
    2
    Yoshi Matsuzaki
    Principal Cloud Support Engineer - Snowflake
    Work:
    Technical Support in English/Japanese (7:3)
    Launching Japanese language support team
    SQL functionality expert
    Escalation review
    Supportability readiness for new features
    Background:
    Infrastructure Engineer (MSP)
    Support Engineer (Cloud Vendor)
    Database Engineer of MySQL engines (Cloud Vendor)

    View full-size slide

  3. © 2020 Snowflake Inc. All Rights Reserved
    SNOWFLAKE?
    3

    View full-size slide

  4. © 2020 Snowflake Inc. All Rights Reserved
    SNOWFLAKE?
    4
    DATA
    SOURCES
    OLTP DATABASES
    ENTERPRISE
    APPLICATIONS
    THIRD-PARTY
    WEB/LOG DATA
    IoT
    DATA
    CONSUMERS
    DATA MONETIZATION
    OPERATIONAL
    REPORTING
    AD HOC ANALYSIS
    REAL-TIME ANALYTICS
    → Data Platform → Data Cloud

    View full-size slide

  5. © 2020 Snowflake Inc. All Rights Reserved
    SNOWFLAKE ARCHITECTURE
    5

    View full-size slide

  6. © 2020 Snowflake Inc. All Rights Reserved
    Snowflake
    6

    View full-size slide

  7. © 2020 Snowflake Inc. All Rights Reserved 7
    Snowflake
    50-500 MB
    ( : S3)
    … Partition Pruning
    /
    NDV ( )

    View full-size slide

  8. © 2020 Snowflake Inc. All Rights Reserved 8

    View full-size slide

  9. © 2020 Snowflake Inc. All Rights Reserved 9
    ( )
    “ ( : immutable)

    - Wikipedia

    View full-size slide

  10. © 2020 Snowflake Inc. All Rights Reserved 10
    : (UPDATE)
    :
    → I/O

    :

    View full-size slide

  11. © 2020 Snowflake Inc. All Rights Reserved 11
    'a' 'b' 'c'
    Table Partitions
    UPDATE tab SET col = 'd'
    WHERE col = 'a';
    'a' 'b' 'c'
    Table Partitions
    'd'
    v1 v2

    View full-size slide

  12. © 2020 Snowflake Inc. All Rights Reserved 12
    I/O
    … OLAP
    OLTP
    OLAP

    View full-size slide

  13. © 2020 Snowflake Inc. All Rights Reserved 13
    / /
    :
    UNDO (MySQL, Oracle)
    UNDO
    (PostgreSQL)
    :

    View full-size slide

  14. © 2020 Snowflake Inc. All Rights Reserved 14
    → /
    → Time Travel
    90 /
    SELECT * FROM tab AT (TIMESTAMP => ...);
    SELECT * FROM tab BEFORE (STATEMENT => );
    CREATE TABLE tab CLONE src AT (TIMESTAMP => ...);
    → Fail Safe
    7
    GC ...

    View full-size slide

  15. © 2020 Snowflake Inc. All Rights Reserved 15
    Table
    SELECT col FROM tab;
    'a' 'b' 'c'
    Partitions
    'd'
    v1 v2
    SELECT col FROM tab
    BEFORE (STATEMENT = ...);
    a
    b
    c
    b
    c
    d

    View full-size slide

  16. © 2020 Snowflake Inc. All Rights Reserved 16
    ( )
    : M:N

    View full-size slide

  17. © 2020 Snowflake Inc. All Rights Reserved 17
    ... /
    Zero Copy Cloning
    Data Sharing, Data Exchange ...

    View full-size slide

  18. © 2020 Snowflake Inc. All Rights Reserved 18
    Zero Copy Cloning
    → Data Sharing ( )
    → Data Exchange (1:1 )
    → Snowflake Data Marketplace ( )
    → Snowflake Data Service Marketplace ( )
    → ...

    View full-size slide

  19. © 2020 Snowflake Inc. All Rights Reserved 19
    → → →…
    (Extensibility)

    View full-size slide

  20. © 2020 Snowflake Inc. All Rights Reserved 20
    Extensibility ( )
    Snowflake
    Snowflake extensible
    → (MP → S/C )
    → (S/C → Zero Copy Cloning)
    → (Cloning → Data Sharing → ...)
    Snowflake

    View full-size slide

  21. © 2020 Snowflake Inc. All Rights Reserved 21
    ... but ( )
    ... but
    ...

    View full-size slide

  22. © 2020 Snowflake Inc. All Rights Reserved 22
    → →
    Snowflake

    View full-size slide

  23. © 2020 Snowflake Inc. All Rights Reserved
    Snowflake

    View full-size slide

  24. 2020 12 10
    11:00 12:00am
    Live
    Snowflake
    Snowflake trocco
    Snowflake primeNumber

    View full-size slide

  25. © 2020 Snowflake Inc. All Rights Reserved

    View full-size slide