A DevOps journey at ABN AMRO @ DevOpsDays oNLine 2020

Recently ABN AMRO embraced their DevOps journey, recognising that they need to continue in the path of continuous improvement. The market demands are high, and they continue to crunch knowledge and create new insights on top of the previous transformations.

Transcript

1. A DevOps journey at ABN AMRO 1 João Rosa (Strategic Software Delivery Consultant) - Xebia
2. Learnings } Provide CLEAR GUIDANCE for teams, with minimum set of requirements } Not the program, but LINE ORGANISATION OWNS the transformation } Set clear MILESTONES to push delivery } Experiment, fail and ADAPT } CONTINUOUS DIALOGUE with 2nd & 3rd line parties } Mindful of culture and SUB-CULTURES – Everyone is unique! } Make it VISUAL and fun, to strive for TRANSPARENCY 2
3. ABN AMRO at a glance 3
4. 4 ABN AMRO Agile transformation done in 2017 A GRID is where blocks are grouped together within the same business area A BLOCK is a small team, that owns a certain part of functionality end-to-end A CIRCLE is a group working within a special subject area or with unique skills A TRIANGLE is a community of members with shared interests
5. Baseline for the transformation 5 Agile Blocks 560 6500 Applications 1.200 Agile Grids People in Grids 47
6. Operating model outlined for moving to Cloud & DevOps 6 1 2 3 4 5 A DevOps block and its PO are responsible for all change & run tasks for the application(s) it manages A DevOps block can independently release functionality into production A DevOps block has an automate everything mindset A DevOps block will use easily consumable standardized services (e.g., infra, security) All team members in a DevOps block contribute to change (80% of their time) and application related run work 10 CADM & CISO set standards for enterprise & security architecture and ensure adherence from blocks (incl. signoff on pipeline and other automation) 7 The tower monitors & signals integrity and currency of app and infra landscape, with intervention mandate for major incidents The helpdesk and bridge execute SOPs defined by the DevOps blocks or route incidents towards the DevOps blocks 8 Scarce expert resources join blocks temporarily in flow-to-work mode 6 DevOps toolchain DevOps DevOps DevOps Tower Security & Architecture Standards Shared services Service catalogue with API consumable infra, platform, DevOps & security services +95% Tailor made <5% SOC Helpdesk Bridge Business grids (distribution, product & enabling) DevOps DevOps DevOps RET DevOps Infra & platform broker Public cloud DevOps DevOps DevOps Tools e.g. monitoring Security & compliance services DevOps DevOps DevOps Specialty infra services Security tools DevOps DevOps DevOps e.g. CMS, Mainframe Infra managed services Infra components The broker sets offering, pricing and SLAs for standardized services (public cloud, on- premise infra, security & CI/CD) 9 Key principles
7. Good building blocks, but also still a lot of detailing to do for the IT Transformation powered by the Apollo program 7 Buy-in of the transformation on ExBo and ExCo level Planning, planning, planning How to cope with the change in a heavily outsourced environment HR and restructuring and process design incl. risk control framework Define our change management approach for teams Engagement of ABN AMRO on all levels and dimensions and facilitate trainings Build the strategic platform incl. engineering system
8. Define what DevOps is at ABN AMRO 8 DEVOPS AT ABN AMRO. DevOps is a way of working that emphasizes collaboration between business, software development and operations. DevOps extends the Agile principles by further streamlining and automating the product lifecycle and enabling cross-functional teams to take ownership of their product from an end-to-end perspective. Keep learning You build it, you run it, you fix it, you own it Automate everything mind- set All team members contribute to change and run work Create flow Use easy consumable and standardized services analyse and prioritize work Backlog management build code Development measure code quality Test Application monitoring Monitoring Events validation of acceptance Change Deploy & release Incidents deploy code into production solve incidents event creation Team autonomy Nail agile Everything as code
9. Minimum viable compliancy to guarantee the autonomy of a DevOps team 9
10. “Beau’s hell” – filtering and aligning capability requirements to a bare minimum but leaving room for inspiration 10 Req_ID Capability Type Requirements Assessment Comments Reference Material Control reference CO_L1_001 Level 1 Control The integrity of the configuration items used for my applications and services is guaranteed by a fully accurate and timely updated Configuration Management Database (CMDB) by my team in ServiceNow. To be filled in CBSP reference information will be shared during CBSP QuickScan sessions. https://intranet.nl.eu.abnamro.com/nl/assets/108-48-20-IT-Configuration-Management-Policy-July2019_tcm582- 1743557.pdf C-00006187 - EC_ISO-04 Application inventory CO_L1_002 Level 1 Control Service recovery plans must be available for CIA rating Availability = 1 and for CIA rating Availability =2 and must be updated at least once a year. Disaster Recovery tests are defined and scheduled for all our application(s) with CIA Availability = 1. Results are registered in DR Dashboard on connections. Disaster Recovery test is performed at least every 12 months for our applications with Recovery Time Objective (RTO) 0-1, and at least every 24 months for all our applications with RTO 2-4. To be filled in CBSP reference information will be shared during CBSP QuickScan sessions. https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=2690c749-1ae7-403f-9a99-a32b6e59fe5e. C-00007725 - Service Recovery Plans are available C-00007726 - Quarterly the BCO verifies that the frequency of DR testing matches the frequency required by the BCM policy C-00007727 - Quarterly the IT-SCM SPoC monitors timely execution of planned DR and , if applicable, follow-up C-00007728 - Quarterly the Business Continuity Officer verifies recording of DR related issues, including recording of proper follow-up CO_L1_003 Level 1 Control All applications owned by my team are registered in the One Application Referential (OAR ). To be filled in https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=b627c98d-7e64-4ed1-8d38-2f8d002ec03a C-00012952 - The application data in the Asset Inventory is complete CO_L1_004 Level 1 Control Block administration is up-to-date with all required information (see Guidelines on connections page). Please pay attention to: - all necessary information on what your block is supporting: owned OAR's, block email, email addresses of team members, phone numbers. - correct administration of your teams DevOps roles as this will define your teams rights in ServiceNow (Product Owner, Scrum Master, IT Engineer etc.) - update AGF with relevant team information. To be filled in https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=3b27a50e-d57c-46ee-9d91-8c5fd94e695a C-00014834 - Quarterly I&A performs a check on the manual part of the JoMoLea process, C-00015173 AWS - IAM setup and Monitoring CO_L1_005 Level 1 Requirement Service Administration in ServiceNow is up-to-date: - All relations of your applications are defined in ServiceNow (upstream & downstream relations) and understood by the entire team. - All stacks/resource groups/Configuration Items are tagged to the correct Business Application of your Business Service - All end users are subscripted to the service, in order to be able to raise calls via de Self-serving portal. (if applicable). To be filled in https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=37ce103e-1701-4ab1-9954-a1a635967946 Not Applicable CO_L1_006 Level 1 Requirement Roles & responsibilities: - Process roles to handle Major and Complex Incidents including the communication via the prescribed channels are formally recognized, defined and assigned in the DevOps team. - Process roles to approve Root Cause Analysis documents and the underlying SIP actions are formally recognized, defined and assigned in the DevOps team. - Segregation of accountability and responsibility between the Product Owner, IT Lead and the DevOps team with regard to the execution of the Incident and Problem process is fully implemented. - Segregation of accountability and responsibility between the Product Owner, IT Lead and the DevOps team and between Dev-engineers & Ops-engineers with regard to the execution of the Change Management process is fully implemented (e.g. There is a single owner who is responsible for assessing Major and Emergency changes). To be filled in Not Applicable CO_L1_007 Level 1 Control Only the central IT service management tooling (ServiceNow) is used for core processes Incident, Problem, Change and Call management. To be filled in https://ibmaabpr.service- now.com/u_published_documents_dms_revision_list.do?sysparm_userpref_module=6a46d7c04f385300feb3d19f0310c75d&sysparm_view=OPS%20Manu al&sysparm_query=dms_type=ee5155444ffc9340a300d2ff0310c797^ORdms_type=d7649d804f30d340a300d2ff0310c7a9^ORdms_type=75e459c04f30d34 0a300d2ff0310c76c^ORdms_type=565595044f30d340a300d2ff0310c74c^ORdms_type=70f5d1c44f30d340a300d2ff0310c754^ORdms_type=4f36d5c44f30d 340a300d2ff0310c7bd^EQ^GROUPBYdms_u_record^ORDERBYdms_type^ORDERBYrev_attachment&sysparm_clear_stack=true C-00010849 Incidents are registered correctly C-00010877 Yearly the effectiveness of automated controls for change management in the Service Management Application is tested CO_L1_008 Level 1 Control On call duty for DevOps team members during and outside office hours is in place for owned critical applications and business services chain(s) with impact (CIA for Availability = 1). To be filled in Will be worked out by Apollo program and published when available. Check this Apollo page: https://social.connect.abnamro.com/wikis/home?lang=nl#!/wiki/W7a3dfeeec2fa_4143_a0dc_1ac023f65e31/page/Organisational%20Design C-xxxxxxxx- to be provided CO_L1_009 Level 1 Control Service Commitments ( e.g. SLA ) are defined for Availability for each application with Availability level 1 for team and vendor performance To be filled in https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=b9928e38-76df-4863-8e69-3c66d72e370a C-00014804 - IT Incident Management resolution times are met CO_L1_010 Level 1 Control For every off-premises application, the team has delivered an exit plan according to the existing ABN AMRO exit strategy. This plan is fitting for use and nature of the concerning application and is approved by responsible DAO and BAO. To be filled in https://social.connect.abnamro.com/wikis/home?lang=nl#!/wiki/W4a15ff48670e_4510_a692_e52743f8cd78/page/Set%20up%20Exit%20strategy C -00015176 AWS - Secure disposal of data, CO_L2_011 Level 2 Requirement When responsibilities have changed (e.g. due to higher maturity or changes in your team) block administration is updated. To be filled in https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=3b27a50e-d57c-46ee-9d91-8c5fd94e695a (also contributes to RCF control: C-00014834 - Quarterly I&A performs a check on the manual part of the JoMoLea process) Not Applicable CO_L2_012 Level 2 Requirement The team knows where to find the change calendar and how to use it to speed up the Root Cause Analyses (e.g. technical analysis) process in case of disturbances To be filled in https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=aa328a19-ed02-49be-801b-c2b6c39d3883 Not Applicable CO_L2_013 Level 2 Requirement To reduce the number and impact of future incidents, Problem Management is used by the team to identify the actual cause of one or more incidents through recurring incident analysis To be filled in https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=1890cce5-6a88-4140-931d-192330bed0ad Not Applicable CO_L2_015 Level 2 Control A status change of a Configuration Item stored in the Service Now configuration management database can only be done following the change management process. To be filled in https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=b325f6cb-c415-4849-80f4-1ab83ed7e255 C-00011327 LC_IT-04 Deltas of the reconciliation between the CMDB and the daily infrastructure scan are discussed and followed up C-00011328 LC_IT-04 Differences between changed CIs and registered CI changes in ServiceNow Blue are discussed with IBM and monitored C-00015869 LC_IT-04 Differences between changed CIs and registered CI changes in CMDB are discussed with DevOps teams CO_L2_016 Level 2 Control Retention/backup services are in place according to RTO - RPO requirements agreed with Business. To be filled in CBSP reference information will be shared during CBSP QuickScan sessions. For AWS refer to: https://social.connect.abnamro.com/wikis/home?lang=en#!/wiki/Wbb310a1c98f8_4ed8_97fb_ed4d14b3a06d/page/Standards%20%26%20Guidelines For IBM environments; refer to TSM in your team. C-00015172 - AWS - Backup and retention of data C-00015177 Azure - Secure disposal of data CO_L2_017 Level 2 Requirement Root cause analysis (RCA) are drawn up on major incidents by all suppliers including Cloud and SAAS Service providers To be filled in Not Applicable CO_L2_018 Level 2 Requirement Knowledge articles in Service Now for user support are created and published To be filled in https://aabsiampr.service- now.com/myit?id=myit_kb_article&sys_id=10ef229ddb29d3480f4416d15b961983&knowledge_base=678ec474db9ddf80bd2c83305b961966 Not Applicable CO_L2_019 Level 2 Requirement Availability, incident and change handling is regularly discussed with stakeholders including Cloud or SAAS-Service providers. To be filled in Not Applicable CO_L3_020 Level 3 Requirement To sustain the required Business level of availability our team uses the Mean Time Between Failure (MTBF) indicator to make reliability improvements for components that have failed after a breakdown and to shorten maintenance and repair time. To be filled in https://social.connect.abnamro.com/wikis/home?lang=nl#!/wiki/W894ba23ada96_4868_883b_d28d07865797/page/D2C%20- %20Detect%20to%20Correct%20-%20Value%20Stream Not Applicable CO_L3_021 Level 3 Requirement An effective capacity management plan, including forecast, for all our used IT components is in place to deliver the highest quality service—at the lowest possible cost. To be filled in CBSP reference information will be shared during CBSP QuickScan sessions. https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=d6a3b434-f3a7-4df2-b6ee- f46d14809ed5&ftHelpTip=true. Not Applicable CO_L3_022 Level 3 Requirement Continual Service Improvement is embedded in the DevOps way of working and improvement initiatives, derived from relevant measurements and KPis, are recorded in Service Now while the actions themselves are put in the Back log. To be filled in https://social.connect.abnamro.com/communities/service/html/communitystart?communityUuid=692949e8-0718-40be-9385-d8b2306b4547 Not Applicable CO_L3_023 Level 3 Requirement The CMDB is automatically updated when changes occur in the IT Landscape To be filled in CBSP reference information will be shared during CBSP QuickScan sessions. Not Applicable What is mandatory and what is an efficiency requirement? What compliancy is referenced? Where can you find detailed information What does my team need to do and how do I score myself on it
11. Channel all the knowledge towards the teams onshore and offshore where required 11 SECURITY WIZARD CI/CD ENABLER PLATFORM SUPPORT MONITOR ENGINEERING SUPPORT RUN SUPPORT INTEGRATOR TRANSFORMATION FACILITATOR MIGRATION Outcomes Grid Journey } Planning in 2 month cycles } 6 CoEs provide experts } One uniform channel } Build-up support offshore Transformation support team ready
13. Internal Start-ups 14
14. Journey Pioneers 15 Photo by Aron Visuals on Unsplash Why DevOps?
15. Journey Pioneers 16 Photo by Zdeněk Macháček on Unsplash Earn trust
16. Journey Pioneers 17 Photo by Kevin Erdvig on Unsplash Meetings as campfires
17. Journey Pioneers 18
18. Journey Pioneers 19
19. Journey Pioneers 20
20. Journey Pioneers 21 Photo by Braden Collum on Unsplash Feedback
21. Results for the grid • Cost optimization with pragmatic approach to business-critical applications • Chain improvements with drop on calls (~x10) per week on call center • Time-to-market reduction • Release anytime for mobile applications • Extreme automation for the software lifecycle • First teams to adopt the new risk and compliance processes • Cloud approval process • Dutch and European Central Banks verification tests 22
22. How was it possible? 23
23. Make it tangible 24
24. Make it tangible 25
25. Make it tangible 26
26. Freedom of engineering 27
27. Learnings } Provide CLEAR GUIDANCE for teams, with minimum set of requirements } Not the program, but LINE ORGANISATION OWNS the transformation } Set clear MILESTONES to push delivery } Experiment, fail and ADAPT } CONTINUOUS DIALOGUE with 2nd & 3rd line parties } Mindful of culture and SUB-CULTURES – Everyone is unique! } Make it VISUAL and fun, strive for TRANSPARENCY 28
28. Q&A 29