Near Real Time Data Warehousing - The Final Frontier?

With the advent of Big Data, does this now mark the end of "traditional" Data Warehousing?

The differences and similarities are highlighted in this presentation and whether or not there is a case for Data Lakes to co-exist with Data Warehouses and Streaming technologies.

Transcript

1. NEAR REAL-TIME DATA WAREHOUSING – THE FINAL FRONTIER?
2. Elffar Analytics Joel Acha Oracle Technologies since 1997 @ElffarAnalytics elffar.co.uk info@elffar.co.uk #obihackers IRC channel 2
3. 1. Data Warehouse Let’s start with a simple definition
4. “ "A data warehouse is a copy of transaction data specifically structured for query and analysis." Ralph Kimball 4
5. 5 Data Integration Methods Traditional ETL CDC Replication Real Time Streaming Batch Based High Latency “Real Time” Low Latency Low Latency In-line in-memory transformation
6. Data Movement ▹ Batching ▹ Micro Batching ▹ Streaming 6
7. Main Characteristics Batching ▹ New data elements grouped into a batch ▹ Based on a time- based batch interval Micro Batching ▹ New data elements more frequently grouped into a batch ▹ Real-time analytics not essential Streaming ▹ Event driven architecture ▹ Low latency is critical 7
8. 2. Big Data Yet another simple definition...
9. “ "larger, more complex data sets, especially from new data sources." source: oracle.com
10. BIG DATA The final nail in the coffin? 10 3 V's: 1. Velocity 2. Volume 3. Variety
11. 3. Data Lake Last definition – I promise...
12. “ "A centralized repository that allows you to store all your structured and unstructured data at any scale." 12 source: amazon.com
13. Big Data – Data Lake ▹ Schema on Read ▹ Data stored in raw form ▹ "Freeform" Are Apples & Pears the same? Data Warehouse ▹ Schema on Write ▹ Structured Data ▹ Query limitations ▹ Value of data clear from the outset 13
14. Complimentary Technologies 14 source: DellEMC.com
15. Use Cases Data Warehouse ▹ Highly curated data ▹ Structured standard reporting Data Lake ▹ Data Scientist access to raw data ▹ Flexible, reactive business model 15
16. THE BEST OF BOTH WORLDS? Streaming ▹Next generation Data Integration ▹In-flight ▹Real-time ▹Can load into data lakes and data warehouses 16
17. THANKS! Any questions? 17 @ElffarAnalytics elffar.co.uk info@elffar.co.uk #obihackers IRC channel