Lakehouse project

Transcript

1. Begoña Villamor Get more value out of your data starting with a good architecture A real project using Oracle Cloud data lakehouse platform
2. JD Edwards Managed Services Business Analytics & Big Data Cloud Netsuite Apps & Mobile Database Technology Solutions 16 10 200 250+ YEARS OFFICES EMPLOYEES GLOBAL CLIENTS
3. Customer requirements Current situation Implemented solution Bank in Central America Users extract data from production databases, performance at risk Applications in: Oracle 10g, 12c, SQL Server, Files, SAAS Creation of a data lake for query off-load Add semi-structured data from external applications and files Creation of a corporate DWH Data lakehouse platform. Data lake: - Object Storage - Autonomous Transaction Processing Datawarehouse: - Autonomous Data Warehouse Data integration: - Oracle Data Integrator Low-code development: - Oracle Application Express Use of ODI automations New secure source for reporting Autonomous database flexibility Business requirements are met quickly Analytics capabilities in DL and DWH
4. Old architecture Transactional Integration Storage Analysis
5. Manual data New architecture ELT Power BI App #1 App #n Excel SQL Transactional Ingestion & Transform Storage Analysis SAAS applications Object Storage DEV DB ... DB
6. Manual data New architecture ODI ATP Power BI App #1 App #n Excel SQL Data Lake Bronze ADW Transactional Ingestion & Transform Storage Analysis SAAS applications Object Storage APEX ATP Integration Silver DWH Gold ...
7. • Security • Performance • Some huge tables +500 GB • Low-maintenance • Low-code • Template based integrations • 10+ data sources. • Different character sets, • Data type conversions. • High latency network. • Execution control and monitoring. • Tables with manual maintenance Project challenges Autonomous databases Oracle Data Integrator - ODI Oracle Application Express - APEX
8. • Database service available in Oracle Cloud • Completely managed by Oracle • Self driving, self securing, self repairing • Based on Oracle 19c • Automatic indexing • Automatic partitioning • Application Continuity • Three flavours based on your main workload • analytic, transaction processing or JSON Autonomous database
9. Autonomous database Easy Fast Elastic
10. Hybrid partitioned tables • Only the most recent data in the big tables is live data. • Oracle 19c hybrid tables store part of the data as external data (files) and part as database table space. • External data can be stored in Object Storage. • Live data (most recent) can be stored in the database space and historic data can be stored in Object Storage (much cheaper). • The access is transparent for applications The "Tables with 500 GB" challenge
11. • Autonomous databases include over 30 ML algorithms implemented as SQL functions • All processing inside the database, no data movement, database security applies • Languages: OML4SQL, OML4Py (Python), OML4R (R) • Interface for data scientist: Oracle Machine Learning Notebooks, based on Zeppelin notebooks • Auto Machine Learning In-database Machine Learning
12. • Reverse engineering for database objects and files • Automatic generation of targets • Automatic generation of simple mappings • Based on code templates • Based on execution contexts: separate physical and logical layers to allow easy environment promotion Oracle Data Integrator - ODI
13. ODI components catalog Powerful database available Agility to add new content Controlled database cost Secure source for reporting BENEFITS FOR THE CUSTOMER Machine Learning ready
14. Combination of a data lake and a data warehouse • From data lakes takes: • Low-cost • Flexible storage • Unstructured data • From data warehouses takes: • Data management features • Data cleansing and ETL tools • Analytical query performance SQL or API access to every layer Easy movement from low-cost storage to analytical storage Artificial intelligence and ML capabilities in every layer Seamless access from the database to object storage Data Lakehouse platform
15. Q&A Quistor Enterprises B.V. Steenbergsestraat 49 4611 TD Bergen op Zoom The Netherlands Quistor Spain Madrid C/ Medea, 4 – 4ºC 28037 Madrid Spain Quistor Central Europe s.r.o. Zirkon Office Center Sokolovská 131/86 186 00 Praha 8 – Karlín Czech Republic Quistor Spain Coruña BCA28 Office 2.5, C/Copérnico, 6, Poligono de A Grela Spain Quistor France SARL 60 avenue Charles de Gaulle 92200 Neuilly Sur Seine France Quistor Spain Malaga Calle San Lorenzo 23, 1º, 29001 Malaga Spain Quistor UK Limited Connect House Kingston Road, Leatherhead KT22 7LT Surrey United Kingdom RTT Srl – Quistor Italy Via Verdi, 4 22070 Rovello Porro Como Italia Quistor BVBA Doornaardstraat 63 2160 Wommelgem Belgium RTT Srl – Quistor Italy Via Campo Farnia, 130 00178 Roma Italia