From Commit to Production: A Journey through a Continuous Delivery Pipeline

A hands-on demonstration of how code travels through the various quality assurance stages at Cytech Mobile, before it reaches the company’s clients. It is a real-world example of how to achieve High Quality, Low-risk releases, and reduce both costs, as well as time-to-market - through Automation - ultimately resulting in better products.

Transcript

1. From Commit to Production A Journey through a Continuous Delivery Pipeline DEVOPS Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
2. From manual labour, to automation Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
3. Cytech Mobile Software Vendor for Telecoms Software: SMSC, Mobile Payments, Mobile Messaging, Mobile Marketing Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
4. 4 15 23 07 Configuration Management Our software, is not just our source code. VERSION Control Never forget anything, ever again. ANTI-PAtterns What we realized was the heart of our problems DEVOPS BASICS Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
5. 5 30 32 27 Deployment pipeline No pipes involved. Automation is. Testing Strategies What to test, How much Why Continuous integration More than just tools: A practice Continuous Integration Continuous Delivery Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
6. 6 36 closing Q & A Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
7. ANTI PATTERNS 7 Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
8. MANUAL DEPLOYMENTS 8 Common Signs: Extensive docs about how to deploy and what to do in every scenario when things go wrong Rely mostly on manual testing to verify app is running correctly Frequent calls to dev team on release day Frequent corrections to the release process, *during* the release Unpredictable releases Roll-backs happen often it’s 2am. You’re still trying to figure out what went wrong. Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
9. MANUAL DEPLOYMENTS 9 Why this is bad: You cannot rule out human error in deployments Manual deployment requires documentation Documentation is expensive to write (collaboration between many ppl), yet VERY easy to become out-of-date Deployments require expertise, yet are boring tasks. Your best people are underutilized. Manual deployments cannot be tested prior to the deployment itself. Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
10. Deploy TO STAGING ONCE DEVELOPMENT COMPLETE 10 Common Signs: System has so far only been tested on developers’ machines Production-like environment is “expensive” Deployment scripts, installers, configuration files, db migrations, etc. are passed on to ops by devs, entirely untested in production-like env. Development team is not working with Operations team on a day-to-day basis Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
11. Deploy TO STAGING ONCE DEVELOPMENT COMPLETE 11 Why this is bad: With the first release come the most bugs. Yet, it is already too close to the deadline. Last-minute patches, fixes, etc. impose a huge communication overhead The bigger the difference between `dev` and `prod` envs, the less realistic the assumptions the devs have to make: e.g. Windows on devs’ laptops, Linux in prod. The longer the release cycle, the harder it is to fix bugs. Developers do NOT remember the specifics of what they were working on 3 weeks ago (the same way they do “now”) Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
12. MANUAL CONFIGURATION MANAGEMENT 12 Common Signs: Even after many successful deploys to staging, deployment into production fails. Servers in a cluster behave differently. Some can serve less load than others. Operations team needs a long time to prepare a new environment for release Inability to go back to a previous known-working configuration (especially to older OS, RDBMS, application server version) System configuration is done by connecting directly (SSH) and modifying files by hand Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
13. MANUAL CONFIGURATION MANAGEMENT 13 INSTEAD! ALL aspects of testing, staging, production (including 3rd party libs) are configured through version control All means: OS, patch levels, OS config, your application stack, its config, infra config, etc. Every change to production recorded (auditable). No manual changes allowed. Only through automated process. Use same automated process to roll-back to previous version. Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
14. FEEDBACK ACTED UPON ASAP FEEDBACK RECEIVED ASAP EVERY CHANGE TRIGGERS FEEDBACK CAN WE DO BETTER? 14 Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
15. v 0.1 VERSION CONTROL 15 v 0.2 v 1.0 v 1.1 v 2.0 Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
16. What TO KEEP IN VERSION CONTROL 16 Source Code, obviously. Unthinkable that your company’s software lives on your developers’ laptops alone. Configuration Any configuration files for your software, including their variations depending on the different environment the software is running on (e.g. for dev / test / staging / prod) Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
17. What TO KEEP IN VERSION CONTROL 17 Dependencies This involves any 3rd-party libraries your software depends on to run. Whether as binaries or (preferrably) through a dependency management framework (e.g. Maven for Java). Environment (i.e. Servers !!! ) OS version. OS Configuration. 3rd-party software: •application servers •web servers •RDBMS •message brokers Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
18. MUST DO’S 18 MANY PER DAY EXPLAIN WHY, NOT WHAT MEANINGFUL MESSAGES REGULAR COMMITS Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
19. PEER REVIEW 19 at least 2 ppl know QA MERGE REQUESTS 4-eye principle Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
20. Infrastructure AS CODE 20 aws AWS CloudFormation TERRAFORM ( HASHICORP ) Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
21. Infrastructure AS CODE 21 Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
22. Infrastructure AS CODE 22 shall we take a look? Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
23. CONFIGURATION MANAGEMENT 23                     Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
24. CM IN A NUTSHELL 24 base image CM TOOL INSTALL & CONFIGURE 3rd party software: * App server * WEB SERVER * RDBMS * MSG BROKER * CACHE chef puppet ansible salt FINAL image environment config: * DEV * test * staging * prod packer
25. What TO INSTALL 25 base image CM TOOL packer
26. HOW TO INSTALL IT 26 chef recipes os commands
27. CONTINUOUS INTEGRATION 27 Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
28. where TO RUN THE TESTS 28 DEV PROD as similar as possible!! Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
29. 29 Jenkins ci server STARTS DOCker container maven:3 `mvn clean package` Unit tests Integration tests FUNCTIONAL tests create docker img compile push docker image to repo How WE RUN THE TESTS payments api docker image Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
30. TESTING STRATEGIES 30 Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
31. WHAT tests do we run 31 run with *EVERY* CHANGE run as often as possible acceptance tests Commit Stage test code works test app works fast (< 5 mins ) take longer in memory whole stack
32. DEPLOYMENT PIPELINE 32 Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
33. 33 How we deploy aws ec2 container service (ECS) Cluster services tasks EC2 Container Service aws ec2 Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
34. 34 Jenkins ci server How we deploy updates appropriate service rolling update deregister from load balancer upload new task definition ecs stop 1 “old version” container launch 1 “new version” container register on load balancer Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
35. hands on
36. The END 36 Thanks !! Cytech Mobile | Yorgos Saslis | @gsaslis | yorgos@cytech.gr
37. Yorgos Saslis Contact me on 37 www.cytechmobile.com twitter.com/gsaslis www.linkedin.com/in/gsaslis yorgos@cytech.gr www.cytechmobile.com Science and Technology Park of Crete, Heraklion, Greece Phone: +30.2810.314127 - Email: info@cytechmobile.com