Success story Datawarehouse Cloud migration

Transcript

1. Rogier van Royen – Business Architect - Rabobank Ralph Broeke – Solution Architect - Rabobank 15th September 2022 Success story Datawarehouse Cloud migration
2. Great journeys are made when you begin We started with our Cloud journey 1,5 years ago 2
3. Context of our Datawarehouse 3 Lending & Insurance Housing Business Lending Finance & Risk DWH-F DIH
4. The setup 4 Started a dedicated team with a vision to move on Cloud to deliver faster business value Involved our stakeholders and partnered with our vendors Defined our way of working Think big start small (MVP) Created a milestone plan with MVP use cases Created our business case The why? and the benefits Create a roadmap with slices of work
5. Consumers On Prem Migration steps and new capabilities Oracle Hubs Internal Consumers Sources Oracle Flat File DB link Flat files WhereScape Scheduler Control Framework
6. Consumers Azure On Prem Migration steps and new capabilities SQL managed instance Oracle D e v O p SQL Hyperscale DWH Hubs Internal Consumers Sources Oracle Flat File DB link Flat files WhereScape Scheduler Control Framework Azure VM WhereScape Scheduler Control Framework
7. Consumers Azure On Prem Migration steps and new capabilities ADF IR DMZ SQL managed instance Oracle D e v O p Azure Data Factory SQL Hyperscale DWH Hubs Internal Consumers Sources Oracle Flat File DB link Flat files JDBC Flat files WhereScape Scheduler Control Framework Azure VM WhereScape Scheduler Control Framework
8. Consumers Azure On Prem Migration steps and new capabilities ADF IR DMZ SQL managed instance Oracle D e v O p Azure Data Factory SQL Hyperscale DWH Hubs Internal Consumers Sources Oracle Flat File SAP HANA Other DB DB link Flat files JDBC Flat files WhereScape Scheduler Control Framework Azure VM WhereScape Scheduler Control Framework
9. Consumers Azure On Prem Migration steps and new capabilities ADF IR DMZ SQL managed instance Oracle D e v O p Azure Data Factory SQL Hyperscale EDL DWH Hubs Internal Consumers Sources Oracle Flat File SAP HANA Other DB DB link Flat files JDBC Flat files WhereScape Scheduler Control Framework Azure VM WhereScape Scheduler Control Framework
10. Lessons Learned & Challenges 10 During our procedure migration for some sprints we had all the data engineers in the team join the working sessions and contribute to the speedy migration of procedures. Teething problems - New environment new technical issues Due to a wrong setting in the Azure DevOps pipeline all our Azure resources got removed With the help of Microsoft team we recovered most of the resources and we recreated some of the resources manually again. This error made us realize the importance of backup and restore options and also disaster recovery options. Realization and importance of having good working sessions within the team Managing expectations internally Managing expectations from internal consumers and business Co-ordination with other departments takes longer Co-ordination with other departments to get things moving e.g. security and risk assessment procedure, implementation of airflow scheduling support etc. Enterprise architecture decision on the datwarehouse delivery and internal consumers Enterprise architecture decision on the datawarehouse delivery and internal consumers Be flexible for additional workload for cloud journey We added more sources to our migration journey WhereScape version upgrade caused two round of upgrades for our cloud program
11. Benefits 11 3. 2018-02-06_Portfolio Classification Setup a clear Cloud vision with objectives Agile - We are able to release when we want and have automated our process Future Ready & scalabale Future proof and scalable - Data warehouse aligned with Rabobank enterprise technology standards Improved Productivity & Cost benefits - Faster development and on demand IT resources to reduce cost Highly motivated and happy employees – Working in the latest technology
12. Feedback & Wrap-up 12
13. Consumers Azure On Prem Migration steps and new capabilities ADF IR DMZ SQL managed instance Oracle D e v O p Azure Data Factory SQL Hyperscale EDL DWH Hubs Internal Consumers Sources Oracle Flat File SAP HANA Other DB DB link Flat files JDBC Flat files WhereScape Scheduler Control Framework Azure VM WhereScape Scheduler Control Framework