Share
Embed
Transcript
74 slides
-
ߊ ࣗѐ - ъਔࢳ • ؘఠ ӝࣿী ҙब ݆ 6֙ର
ূפয • () SaaS झఋস, য়झఋੌ, ܻ٣ • അח Ү פח ... -
दೞӝ ী • п ӝࣿী ೠ ಣоח ୭ೠ ёҙਵ۽ ೮݅,
ҙ ো ী ٜযщਸ ࣻب णפ. • ח AWSח ೂࠗೞѱ ॄࠁও݅, ܲ ӝٜࣿ ۽؋࣌ীࢲ ਊ೧ࠁ ঋই ੜޅػ ࠁо ਸ ࣻ णפ. ೞ݅ ୭ࢶਸ ೧ ৢ߄ܲ ࠁܳ ਵ۰ ೮णפ. 😃 (ೞ݅ trial ೧ࠁӟ ೮णפ) • ৈ۞ ѐ֛ٜী ೧ ݅ ߑधਵ۽ ࢸݺೞח ࠗ࠙ חؘ, ࢎۈ݃ ࢤп ܳ ࣻ ਵ۽ ߉ইٜৈदݶ хࢎೞѷणפ. -
য়ט ࣗѐܾ٘ ղਊ • ؘఠ ۖಬ җѢ৬ അ • নೠ
ۄ٘ ؘఠ ۖಬ ಁ۞ ಝࠁӝ • ରࣁ ؘఠ ۖಬ ӝࣿ ಝࠁӝ - Apache Iceberg, Delta Lake • ݛ ߂ Q&A -
Ӓېࢲ ؘఠ ۖಬ ޥؘ? ই নೠ ഋక ؘఠܳ (ഋ, ࠺ഋ,
߈ഋ) ষաѱ ݆ ೡ ࣻ Ҋ (PB+) -
Ӓېࢲ ؘఠ ۖಬ ޥؘ? ই নೠ ഋక ؘఠܳ (ഋ, ࠺ഋ,
߈ഋ) ষաѱ ݆ ೡ ࣻ Ҋ (PB+) ղ ੋղबਸ ӓೞ ঋח ࢶীࢲ ؘఠ ઑഥо оמ೧ঠೞҊ -
Ӓېࢲ ؘఠ ۖಬ ޥؘ? ই নೠ ഋక ؘఠܳ (ഋ, ࠺ഋ,
߈ഋ) ষաѱ ݆ ೡ ࣻ Ҋ (PB+) ղ ੋղबਸ ӓೞ ঋח ࢶীࢲ ؘఠ ઑഥо оמ೧ঠೞҊ ࢎղ ҳࢿਗ ݽفо زदী ؘఠܳ ઑഥ೧ب ੜ ઑഥغযঠ ೞҊ -
Ӓېࢲ ؘఠ ۖಬ ޥؘ? ই নೠ ഋక ؘఠܳ (ഋ, ࠺ഋ,
߈ഋ) ষաѱ ݆ ೡ ࣻ Ҋ (PB+) ղ ੋղबਸ ӓೞ ঋח ࢶীࢲ ؘఠ ઑഥо оמ೧ঠೞҊ ࢎղ ҳࢿਗ ݽفо زदী ؘఠܳ ઑഥ೧ب ੜ ઑഥغযঠ ೞҊ ۧѱ ࡳইմ ؘఠܳ दпചೡ ࣻ যঠ ೞݴ -
Ӓېࢲ ؘఠ ۖಬ ޥؘ? ই নೠ ഋక ؘఠܳ (ഋ, ࠺ഋ,
߈ഋ) ষաѱ ݆ ೡ ࣻ Ҋ (PB+) ղ ੋղबਸ ӓೞ ঋח ࢶীࢲ ؘఠ ઑഥо оמ೧ঠೞҊ ࢎղ ҳࢿਗ ݽفо زदী ؘఠܳ ઑഥ೧ب ੜ ઑഥغযঠ ೞҊ ۧѱ ࡳইմ ؘఠܳ दпചೡ ࣻ যঠ ೞݴ ղо ਗೞח ؘఠܳ ࡅܰѱ ੜ ਸ ࣻ যঠ ೞҊ ٸ۽ח যڃ ؘఠח ࠁউ ޙઁ۽ ࠁ ঋѢա ੌࠗ݅ ࠁৈঠೞҊ ࢎղ ਃೠ ؘఠח ই ੜ ୶࢚ചغয ܻغয যঠ ೞҊ ؘఠܳ ݅٘ח ۽ਸ ೠ ׀ী ঌইࠅ ࣻ Ҋ ӒѦ ࠺ਊਵ۽ ਬࠁࣻ ೡ ࣻ ਵݴ ղо ృࢎ೧ب ؘఠ ۖಬ غח ؘח ޙઁо হҊ оәݶ पदрਵ۽ ࠅ ࣻ যঠ ೞҊ ѐߊо ইצ ࢎۈب ಞೞѱ ࠅ ࣻ যঠ ೞחؘ Ӕؘ য૰ٚ ࠺ਊ ژ ۴೧ঠೠ... -
Ӓېࢲ ؘఠ ۖಬ ޥؘ? ই নೠ ഋక ؘఠܳ (ഋ, ࠺ഋ,
߈ഋ) ষաѱ ݆ ೡ ࣻ Ҋ (PB+) ղ ੋղबਸ ӓೞ ঋח ࢶীࢲ ؘఠ ઑഥо оמ೧ঠೞҊ ࢎղ ҳࢿਗ ݽفо زदী ؘఠܳ ઑഥ೧ب ੜ ઑഥغযঠ ೞҊ ۧѱ ࡳইմ ؘఠܳ दпചೡ ࣻ যঠ ೞݴ ղо ਗೞח ؘఠܳ ࡅܰѱ ੜ ਸ ࣻ যঠ ೞҊ ٸ۽ח যڃ ؘఠח ࠁউ ޙઁ۽ ࠁ ঋѢա ੌࠗ݅ ࠁৈঠೞҊ ࢎղ ਃೠ ؘఠח ই ੜ ୶࢚ചغয ܻغয যঠ ೞҊ ؘఠܳ ݅٘ח ۽ਸ ೠ ׀ী ঌইࠅ ࣻ Ҋ ӒѦ ࠺ਊਵ۽ ਬࠁࣻ ೡ ࣻ ਵݴ ղо ృࢎ೧ب ؘఠ ۖಬ غח ؘח ޙઁо হҊ оәݶ पदрਵ۽ ࠅ ࣻ যঠ ೞҊ ѐߊо ইצ ࢎۈب ಞೞѱ ࠅ ࣻ যঠ ೞחؘ Ӕؘ য૰ٚ ࠺ਊ ژ ۴೧ঠೠ... 😰 -
Ӓېࢲ ؘఠ ۖಬ ޥؘ? ই নೠ ഋక ؘఠܳ (ഋ, ࠺ഋ,
߈ഋ) ষաѱ ݆ ೡ ࣻ Ҋ (PB+) ղ ੋղबਸ ӓೞ ঋח ࢶীࢲ ؘఠ ઑഥо оמ೧ঠೞҊ ࢎղ ҳࢿਗ ݽفо زदী ؘఠܳ ઑഥ೧ب ੜ ઑഥغযঠ ೞҊ ۧѱ ࡳইմ ؘఠܳ दпചೡ ࣻ যঠ ೞݴ ղо ਗೞח ؘఠܳ ࡅܰѱ ੜ ਸ ࣻ যঠ ೞҊ ٸ۽ח যڃ ؘఠח ࠁউ ޙઁ۽ ࠁ ঋѢա ੌࠗ݅ ࠁৈঠೞҊ ࢎղ ਃೠ ؘఠח ই ੜ ୶࢚ചغয ܻغয যঠ ೞҊ ؘఠܳ ݅٘ח ۽ਸ ೠ ׀ী ঌইࠅ ࣻ Ҋ ӒѦ ࠺ਊਵ۽ ਬࠁࣻ ೡ ࣻ ਵݴ ղо ృࢎ೧ب ؘఠ ۖಬ غח ؘח ޙઁо হҊ оәݶ पदрਵ۽ ࠅ ࣻ যঠ ೞҊ ѐߊо ইצ ࢎۈب ಞೞѱ ࠅ ࣻ যঠ ೞחؘ Ӕؘ য૰ٚ ࠺ਊ ژ ۴೧ঠೠ... ରӔରӔ ಝࠁӝ -
Ӓېࢲ ؘఠ ۖಬ ޥؘ? ই নೠ ഋక ؘఠܳ (ഋ, ࠺ഋ,
߈ഋ) ষաѱ ݆ ೡ ࣻ Ҋ (PB+) ղ ੋղबਸ ӓೞ ঋח ࢶীࢲ ؘఠ ઑഥо оמ೧ঠೞҊ ࢎղ ҳࢿਗ ݽفо زदী ؘఠܳ ઑഥ೧ب ੜ ઑഥغযঠ ೞҊ ۧѱ ࡳইմ ؘఠܳ दпചೡ ࣻ যঠ ೞݴ ղо ਗೞח ؘఠܳ ࡅܰѱ ੜ ਸ ࣻ যঠ ೞҊ ٸ۽ח যڃ ؘఠח ࠁউ ޙઁ۽ ࠁ ঋѢա ੌࠗ݅ ࠁৈঠೞҊ ࢎղ ਃೠ ؘఠח ই ੜ ୶࢚ചغয ܻغয যঠ ೞҊ ؘఠܳ ݅٘ח ۽ਸ ೠ ׀ী ঌইࠅ ࣻ Ҋ ӒѦ ࠺ਊਵ۽ ਬࠁࣻ ೡ ࣻ ਵݴ ղо ృࢎ೧ب ؘఠ ۖಬ غח ؘח ޙઁо হҊ оәݶ पदрਵ۽ ࠅ ࣻ যঠ ೞҊ ѐߊо ইצ ࢎۈب ಞೞѱ ࠅ ࣻ যঠ ೞחؘ Ӕؘ য૰ٚ ࠺ਊ ژ ۴೧ঠೠ... ରӔରӔ ಝࠁӝ Distributed Storage Distributed Query Engine Visualization Discovery Security, Governance (Near) Realtime Data Pipeline, CDC Less-code; e.g.) Amplitude, DBT, SQL Software Engineering, IaC, DataOps Data Type, Compression, Tiering, Cache, Partitioning -
Ӓېࢲ ؘఠ ۖಬ ޥؘ? ই নೠ ഋక ؘఠܳ (ഋ, ࠺ഋ,
߈ഋ) ষաѱ ݆ ೡ ࣻ Ҋ (PB+) ղ ੋղबਸ ӓೞ ঋח ࢶীࢲ ؘఠ ઑഥо оמ೧ঠೞҊ ࢎղ ҳࢿਗ ݽفо زदী ؘఠܳ ઑഥ೧ب ੜ ઑഥغযঠ ೞҊ ۧѱ ࡳইմ ؘఠܳ दпചೡ ࣻ যঠ ೞݴ ղо ਗೞח ؘఠܳ ࡅܰѱ ੜ ਸ ࣻ যঠ ೞҊ ٸ۽ח যڃ ؘఠח ࠁউ ޙઁ۽ ࠁ ঋѢա ੌࠗ݅ ࠁৈঠೞҊ ࢎղ ਃೠ ؘఠח ই ੜ ୶࢚ചغয ܻغয যঠ ೞҊ ؘఠܳ ݅٘ח ۽ਸ ೠ ׀ী ঌইࠅ ࣻ Ҋ ӒѦ ࠺ਊਵ۽ ਬࠁࣻ ೡ ࣻ ਵݴ ղо ృࢎ೧ب ؘఠ ۖಬ غח ؘח ޙઁо হҊ оәݶ पदрਵ۽ ࠅ ࣻ যঠ ೞҊ ѐߊо ইצ ࢎۈب ಞೞѱ ࠅ ࣻ যঠ ೞחؘ Ӕؘ য૰ٚ ࠺ਊ ژ ۴೧ঠೠ... য়ט ݫੋ ઁ Distributed Storage Distributed Query Engine Visualization Discovery Security, Governance (Near) Realtime Data Pipeline, CDC Less-code; e.g.) Amplitude, DBT, SQL Software Engineering, IaC, DataOps Data Type, Compression, Tiering, Cache, Partitioning -
ؘఠܳ ా೧ ࢲ࠺झ, ࠺ૉפझ җѢ৬ അ, ېܳ ঌ ࣻ
নೠ ৬ ా҅ (ݒ, AU, ...) ؘఠܳ ాೠ ࢎѾ ؘఠ ӝ߈ ࢲ࠺झ (Ѩ࢝, ୶ୌ, ੋӝب, ۘఊ, , A/B పझ, ஏ, ...) ... -
ؘఠ ۖಬ ৈ۞ ಁ۞ Data Warehouse Data Lake Data
Lakehouse? DB w/ Distributed Storage, Query Engine Distributed Storage / Distributed Query Engine / Metastore Data Warehouse & Data Lake -
Pros of Data Lake • ইޖ ؘఠա ݄ ೞݶࢲ ॶ
ࣻ (ELT) -> image, audioب оמ! • ߹ب ingestion җ ਃ হ: storageী ೞݶ • Storageо SSoTо غӝ ٸޙী, ೂࠗೠ storage ӝמ ࢎਊ оמ + ੑݍী ݏח query engine ࢎਊ оמ (SQL, API, Code, Framework, External readܳ ਗೞח নೠ ઁಿٜ) • ࠗ࠙ ۄ٘ח storageо ઁੌ ۴ೞ! ژೠ, tieringب оמೞ • External, schema-on-read ߑध ࠺Ү ਬোೞҊ programmatic APIܳ ਗೞח metadata system: Hive, Glue, Dataproc Metastore ١ -
Cons of Data Lake • Underlying storage ઁডਸ Ӓ۽ ਊ߉ח
(e.g. EC, rename, ACID, streaming, ...) • ࢎਊо storageܳ যڌѱ ࢎਊೞוջী ٮۄ ࢿמ ୌର݅߹: data type, format, compression, directory structure, block size, ...) • Lakeܳ ҳࢿೞח componentо ցޖ ݆: storage, query engine, metastore, ... • ؘఠ ҙܻо ࢚ਵ۽ ؊ য۵ - Data swamp • External schema ҙܻо ࠂೞ - file format߹۽, store ߹۽, query engine ߹۽ ܰҊ schema evolution ࠺оੋ ҃ب . • SQL ݅ਵ۽ ؘఠ ҙܻܳ ೡ ࣻ হ - Update, Delete? -
Pros of Data Warehouse • ࢎਊо न҃ॶ Ѫ ݆ হ:
ingestion ೞҊ, աݠח ࠗ DWо ঌইࢲ ೧Ѣ ա ࢎਊо DW ӝמਸ ਊ೧ࢲ customize ೞݶ ػ. • Internal metadata (schema-on-write) ӝ ٸޙী schema evolution э Ѫ ࠺Ү ਬ܂. • ݅ ت ٜ ࣻ . ౠ োٜ ܳ զܻҊ ࢜۽ ݅٘ח ҃о • ױࣽ query engine ࢚ ӝמٜਸ ઁҕೞח ҃о : CDC, SQL ޙߨ ١ • Component ѐࣻо : BigQuery -> ਬࠁࣻ ࠺ਊ Lakeࠁ ਸ ࣻ • SQL native۽ ؘఠܳ ҙܻೡ ࣻ : Update, Delete ١ (ױ, ز ߑधী ٮۄ ࠺ਊ ҳغѢա ਗೞ ঋਸ ࣻ ) -
Cons of Data Warehouse • Ingestion җ . • DWীࢲ
ਗ೧ ঋח ӝמ ࢎਊೡ ࣻ হ: image, audioܳ BigQueryীࢲ ܖ Ҋ रݶ? ղо ࢎਊೞח DWо semi-structured formatਸ ਗೞ ঋחݶ? connectorо ਗغ ঋחݶ? • ইޖѢա ݄ ֍Ҋ ࢎਊೞӝ ࠗझۣ. (оѺ...) • Component ߹۽ ؘఠо ಞചػ: real-time BI servingਸ ਤ೧ࢲח BigQuery ݅ਵ۽ח ࠗೞ. • ೠ ߣ ٜযр ؘఠܳ ߄Ӵਵ۽ ࡐӝ য۵: $$$ • Lakeী ࠺೧ ࠺व оמࢿ ֫. (ౠ multi-cloud ۄݶ) -
Price Comparison: Athena, BigQuery, Snowflake • 1PB JSONਸ ingestion ೠҊ
о, ਘ ௪ܻ 100TB • 1PB JSONਸ Parquet + zstd ୷ೞݶ ۚ 90% ب ୷ܫ ա১ (=100TB) • ۄ٘ ҃ э ۄ٘ ղ э regionীࢲ networkܳ ਊೠҊ о (no public egress) • э region ইפۄݶ $$$ • ࠺ਊ ӝળ ݽف Seoul regionਸ ӝળਵ۽ ೣ • ਘ߹ ࠺ਊਸ ҅ • ଵҊਊੑפ! (ݒ ࠗഛೡ ࣻ ) -
Price Comparison: Athena, BigQuery, Snowflake Athena BigQuery Snow fl ake
Ingestion Free Free Free? Storage 100TB / 0.025$ per GB = $2,500 1PB (uncompressed) / 0.023$ per GB = $23,000? 100TB / 0.025$ per GB = $2,500 + another $2,500 for storage Query 100TB / 5$ per TB = $500 100TB / 6$ per TB = $600 18ݺ ੌҗदрী ॳݶ = $5,952 Total $3,000 $23,600 $10,952 -
ؘఠ ۨ ױਸ ӓࠂೡ ࣽ হਸө? • Underlying storage ઁডਸ
Ӓ۽ ਊ߉ח (e.g. EC, rename, ACID, streaming, ...) • ࢎਊо storageܳ যڌѱ ࢎਊೞוջী ٮۄ ࢿמ ୌର݅߹: data type, format, compression, directory structure, block size, ...) • Lakeܳ ҳࢿೞח componentо ցޖ ݆: storage, query engine, metastore, ... • ؘఠ ҙܻо ࢚ਵ۽ ؊ য۵ - Data swamp • External schema ҙܻо ࠂೞ - file format߹۽, store ߹۽, query engine ߹۽ ܰҊ schema evolution ࠺оੋ ҃ب . • SQL ݅ਵ۽ ؘఠ ҙܻܳ ೡ ࣻ হ - Update, Delete? -
ژೠ ౠ ױٜ Warehouseীب ઓೠ • Underlying storage ઁডਸ Ӓ۽
ਊ߉ח (e.g. EC, rename, ACID, streaming, ...) • ࢎਊо storageܳ যڌѱ ࢎਊೞוջী ٮۄ ࢿמ ୌର݅߹: data type, format, compression, directory structure, block size, ...) • Lakeܳ ҳࢿೞח componentо ցޖ ݆: storage, query engine, metastore, ... • ؘఠ ҙܻо ࢚ਵ۽ ؊ য۵ - Data swamp • External schema ҙܻо ࠂೞ - file format߹۽, store ߹۽, query engine ߹۽ ܰҊ schema evolution ࠺оੋ ҃ب . • SQL ݅ਵ۽ ؘఠ ҙܻܳ ೡ ࣻ হ - Update, Delete? -
Features of Delta Lake / Iceberg • Support Full Schema
Evolution with more expressive DML • Update, Delete, Merge, or by yourself (using Spark, Python, API, ...) -
Features of Delta Lake / Iceberg • Support Full Schema
Evolution with more expressive DML • Update, Delete, Merge, or by yourself (using Spark, Python, API, ...) • Time Travel & Rollback (for reproducible query, by snapshot) -
Features of Delta Lake / Iceberg • Support Full Schema
Evolution with more expressive DML • Update, Delete, Merge, or by yourself (using Spark, Python, API, ...) • Time Travel & Rollback (for reproducible query, by snapshot) • ACID Transactions, Optimized Streaming, Audit Logs, Caching, Data Layout Optimization (Z-order, multiple column ordering), Support ML Features • Open Sources! -
Features of Delta Lake / Iceberg • Support Full Schema
Evolution with more expressive DML • Update, Delete, Merge, or by yourself (using Spark, Python, API, ...) • Time Travel & Rollback (for reproducible query, by snapshot) • ACID Transactions, Optimized Streaming, Audit Logs, Caching, Data Layout Optimization (Z-order, multiple column ordering), Support ML Features • Open Sources! -
࠺ૉפझ ࢚ടী ݏח ӝࣿ ࢶఖ ਃೞ. Ӓ۞۰ݶ ӝࣿী ೠ ೂࠗೠ
೧о ਃೞ. -
Delta Lake & Iceberg ۽؋࣌ ॄب غաਃ? • Delta Lake
҃ ݆ Ҋёࢎীࢲ ࢎਊ . (ೠҴ ੜ ݽܰѷ֎ਃ) • Databricks৬ ҅ডਸ ݛҊ ࢎਊೞח ҃о ݆. য়ࣗझܳ بੑೠ Ҕ ח ݽܰ ѷणפ. • Icebergب ೠҴ ੜ ݽܰѷਵա Spark, Snowflake ӝ߈ ৻Ҵ ഥࢎٜ ݆ ࢎਊೞҊ णפ. -
Redshiftח റоਃ? • Redshiftח DWӟ ೞ݅, OLAPী ؊ оӰҊ
ࠅ ࣻ णפ. • Druid, Clickhouse ۢ... • ખ ؊ ౠࣻ ݾী ࢎਊೞח ѱ જਸ Ѫ эणפ. • e.g. CDC from Postgres • e.g. ~ms latency dashboard -
द҅ৌ, Ӓې ؘఠب ؘఠ ۖಬী ֍ਸ ࣻ աਃ? • द҅ৌ
• ޛۿ݅, InfluxDB э Ѫਸ ӝೞदݶ উ ؾפ. • Ӓې • द ֍ਸ ࣻ ݅, Ӓې ഋకܳ row۽ ಽযࢲ ֍ח Ѫ જणפ. (..) -
ElasticSearchܳ ؘఠ ۖಬਵ۽ ॄب غաਃ? • ߈ ݏҊ, ߈ ইפ
(ѐੋੋ ࢤп) • ESب cluster ҳઑ distributed storage / query engine ਸ оӟ ೮݅, ӏݽ ؘ ఠܳ ࡅܰѱ ઑഥೞח Ѫࠁ ܲ Ѫী ъ ݆ • e.g.) time-series indexing, very fast latency ١ • ݫੋ ؘఠ ۖಬਵ۽ח ࠗೞ݅ ౠ use-caseܳ ࠁ৮ೞח ਊب۽ח 👍