A Devops Journey - Setting up a CI Stack

Continuous integration and continuous delivery/deployment capabilities is a godsend for every software development team. Due to a tight delivery timeline for a project, we had no choice but to implement this critical release engineering capability to enable faster time-to-market.

We will be talking about
• A brief introduction to the concepts of CI/CD
• About our journey and how we landed up with a stack based on Git/Atlassian Stash, Jenkins, Gradle, Nexus
• The challenges and pain points in the DBS environment and what is the preparatory and redtape work required
• Our setup and how we managed to install, configure and test our stack up and running in 3 solid nights
• Brief Demos on
o How to manage POCs through Virtualbox/Vagrant
o The stack and a sample build workflow
• Where we are now in the area of release engineering and our future plans towards Devops nirvana
• How your team can go about go about starting your own devops efforts and how it can align to the DBS ITSS Enterprise Devops efforts

Transcript

1. A Devops Journey: Setting up a CI Stack By Steve Teo/DBS ITT Processing Systems
2. Synopsis Intro to CI
3. Synopsis Intro to CI Our Journey + Demos
4. Synopsis Intro to CI Our Journey + Demos Future Devops Plans
5. Continous Integration (CI) The practice, in software engineering, of merging all developer working copies with a shared mainline several times a day.
6. CI, Visually h"p://blog.jki.net/wp-­‐content/uploads/2012/09/jki-­‐con<nuous-­‐integra<on-­‐graphic-­‐091812.png  
7. CI, Visually h"p://www.programmingresearch.com/wp-­‐content/uploads/2012/11/jenkins-­‐plugin-­‐diagram-­‐saci.png  
8. CI Best Practices Maintain a code repository Automate the build Make the build self-testing Everyone commits to the baseline every day Every commit (to baseline) should be built Keep the build fast Test in a clone of the production environment Make it easy to get the latest deliverables Everyone can see the results of the latest build Automate deployment
9. I have CI, what now? Continuous Integration Continuous Delivery Continuous Deployment
10. Build-deploy workflow h"p://akquinetblog.files.wordpress.com/2012/01/big_picture1.png  
11. Cont. Delivery vs Cont. Deployment h"p://blog.crisp.se/wp-­‐content/uploads/2013/02/con<nuous-­‐delivery-­‐deployment-­‐sm.jpg  
12. ‘Holy Bible’
13. Background DBS + Unnamed Vendor jointly developing new Project using Agile + Devops approach
14. Where we were ✔ Codebase under SCM ✖ CI Capability ✖ Remote Deployment
15. Proposed CI Stack
16. Proposed CI Stack– Why? Project timeline too tight, any deviation from Vendor’s requirements might have affected delivery timeline Can tap on Vendor’s expertise Aligned mostly to upcoming enterprise standards Prior experience and proven model Easier transition to enterprise stack as we would have acquired the necessary knowhow by then
17. Challenges Non-standard software (in DBS Context) Infrastructure not owned by us Servers not ready Firewall Issues Lack of knowledge Lack of precedence Uncertain abt. Enterprise Direction
18. Challenges Non-standard software (in DBS Context) •  Uncertain who can approve •  Uncertain whether it will be approved -> Solution: DBS Non-Standard Technology Approval Website
19. Challenges Infrastructure not owned by us •  No root access to install stuff Servers not ready (then) •  Uncertain delivery timeline •  Uncertain server configuration •  Tighter timeline with every delay -> Solution: Run a POC using virtual machine to familiarize
20. Challenges Firewall Issues •  Uncertain about connectivity between servers in different network zones •  No internet access for build server means having to be manual in so many ways -> No solution so far, manual workarounds needed (but will be optimized)
21. Challenges Lack of knowledge Lack of precedence Uncertain abt. Enterprise Direction -> Solution: Ask ITSS for guidance and updates
22. Demo Time •  Using Vagrant and Virtual Box to create reproducible Proof of Concept/ development environments
23. Proposed CI Stack Architecture #1 DBS Developers Build Machine (On SIT VM) Stash Jenkins Nexus One Way SSL
24. Proposed CI Stack Architecture #2 DBS Developers Build Machine (On SIT VM) Stash Jenkins Nexus One Way SSL Apache HTTP Server (Reverse Proxy) Unencrypted
25. During Review • “If it’s worth doing, it’s worth overdoing it” – Ray Goh
26. Final Architecture DBS Developers Build Machine (On SIT VM) Stash Jenkins Nexus One Way SSL Apache HTTP Server (Reverse Proxy) One Way SSL
27. Mission Impossible •  Infrastructure delays left us with only 3 days to set up •  Due to go for reservist the following week – Do or die mission •  POC and a solid understanding of SSL really helped us a lot
28. How it is being used today DBS Developers Build Machine (On SIT VM) Stash Jenkins Nexus One Way SSL Apache HTTP Server (Reverse Proxy) One Way SSL Vendor Developers
29. Build-deploy workflow h"p://akquinetblog.files.wordpress.com/2012/01/big_picture1.png   1.  We  use  event  hooks  to  inform   CI  server  that  a  commit  has  been   commi"ed   2.  Transfer  and  execute  a  gradle  deploy  script  
30. Environments/Pipelines DEV   SIT   UAT   Staging   1.  SSH  Gradle   Deployment  Script   and  executes  it   2.  Gradle  script   retrieves  ar<facts  
31. Demo Time •  A walkthrough of our CI stack and a sample commit-build workflow
32. Retrospective
33. Where we were ✔ Codebase under SCM ✖ CI Capability ✖ Remote Deployment
34. Where we are now ✔ Codebase under SCM ✔ CI Capability ✔ Remote Deployment
35. Advice on starting up Learn your way around DBS infrastructure and processes Read and experiment (A LOT!) Make small and steady steps – go for the quick wins. Don’t go for overly ambitious stack
36. It’s not just the tools Setup and installation is just the easy part What we can’t change overnight is culture and knowledge Start adopting a devops mindset today!
37. Future Devops Plans
39. Enterprise Stack Switch to enterprise stack once it is ready and viable for our needs •  Releases can steamroll all the way to production through upcoming release automation •  Shortens time-to-market •  Static analysis capabilities and advanced testing suites Temporary integration to existing RMF for Production rollout in the meantime
40. Orchestration Orchestration describes the automated arrangement, coordination, and management of complex computer systems, middleware, and services.
41. Orchestration 21 servers to configure and manage now - painful Solution: Orchestration tools such as Chef, Puppet, Ansible or Salt
42. Chatops Conversation-driven development, pioneered by Github Hubot perhaps?
43. Before I end Some words of thanks • My team members • DBS ITSS Devops • Unnamed Vendor
44. Thank you Any Questions? M: Steve Teo E: xylancer@gmail.com T: @kelltainer