2017 - Natalie Serebryakov - Continuous Deployment

Description
The presentation will be helpful for engineers, who developed an application in Python, or library or microservice. The presentation will cover the practices automating continuous integration, continuous delivery, and continuous deployment with Python ( examples: Jenkins and Gocd).

Abstract
Continuous Deployment is a process to ensure that code can be rapidly and safely deployed to production by delivering every change to a production-like environment and ensuring business applications and services(microservices) function as expected.

Bio
My name is Natalie; I have a Networking and Security background but recently decided to challenge myself in different space. That's why currently I work at Vevo as Tools Engineer in coolest DevOps team in San Francisco. Where part of my job includes building and automating various tools that help our development team be more productive and happy. I have an MS in Compute Since degree. For my first job, I had to do a lot of scripting for configuring and managing thousands of switches, routers and other networking devices in our company testing lab. After using sed, awk, Tcl I discovered and learned Python. It was straightforward to pick up and worked well since then Python became my language of preference. I also like Python for its big community.

In my spare time, I like to spend time with my pup Chai, bake and blog about baking.

Thank you, organizers, for giving me and people the opportunity to speak and this conference and share our knowledge with community .

Transcript

1. Continuous Deployment with Python Natalie Serebryakova PyBay2017
2. About me Software Engineer (Tools) DevOps Team, Vevo LLC Love to get hands dirty on automating tasks and processes natalieserebry@gmail.com
3. Agenda • What is Continuous Delivery • Why and how CD should be implemented • Delivery pipeline • Improvements
4. What is Continuous Delivery ? Continuous delivery (CD) is a capability to continually deploy software to production as frequent as needed. The goal of Continuous Delivery is to enable a constant flow of changes into production via an automated software production line.
5. Why should we start doing this ? • Improve speed to market • Ensured quality • Deploying code is seamless, no longer strenuous • Improves your disaster recovery/business continuity • Improve business agility
6. Implementing Continuous Delivery In our Continuous Delivery Pattern, we implemented two architectural attributes: • testability (“passed” on every environment) • deployability ( low-risk to “build” on every environment) Designing for testability and deployability starts with ensuring our services and modules are composed of loosely-coupled, well-encapsulated components follow the principles of SOLID design
7. SOLID (object-oriented design) In computer programming, the term SOLID is a mnemonic acronym for five design principles intended to make software designs more understandable, flexible and maintainable. (Wiki)
8. Why to automate? • It’s easy with python, many modules available: gocd, jenkinsapi, python-jenkins, docker-py, pykube, etc • save time, stop repeat the same things (DRY) • the complete delivery pipeline is in git: you have predictable recovery because, you know what is happening • machines do it just better • provide tools Development team maintain and can refactor their deployment • no UI (no “you just have to click on”....)
9. How to automate. Real world example The goal in allow Development Team run the single command to: • Scaffold new project (git, provision in the CD tool GoCD, Jenkins) • Checkout it from the source code • Run automated tests • Deploy to every environment
10. I have working python code, how do I start now? Database (PostrgreSQL) App (Flask Gunicorn) Static Assets Webserver (NGINX) TCP Socket Browser HTTP Layout of a non-trivial Flask application Single Server
11. I have working python code, how do I start now? Browser Running many instances of the app, talking to a single DB Application Server 1 App (Flask Gunicorn) Application Server 2 App (Flask Gunicorn) Application Server 3 App (Flask Gunicorn) Static Assets Webserver (NGINX) HTTP Database (PostrgreSQL) TCP Database Server Load Balancer HTTP HTTP HTTP TCP TCP
12. The application running on an abstract collection of resources in production environment Browser primitives such as persistent storage and load balancers that are common to any non-trivial Node 2 App (Flask Gunicorn) App (Flask Gunicorn) Node 3 App (Flask Gunicorn) Content Delivery Network (Static Assets) Load balancer HTTP Database (PostrgreSQL) TCP Node 1 HTTP HTTP HTTP TCP TCP Persistent Disk HTTP the primary pieces we care about (the application components)
13. Architecture using Deployment pipeline The key pattern introduced in Continuous Delivery is the deployment pipeline Deploy Staging Deploy Production Deploy Development Build Test
14. The pipeline breaks down the software delivery process into stages. Each stage is aimed at verifying the quality of new features from a different angle to validate the new functionality and prevent errors from affecting your users. The pipeline should provide feedback to the team and visibility into the flow of changes to everyone involved in delivering the new feature/s. The Continuous Delivery pipeline
15. Proposed solution from the slide 9 Development Team should run the single command • Scaffold new project (git, provision in the CD tool GoCD) • Checkout it from the source code • Run automated tests • Deploy to every environment
16. Components for automated deploys • Python package (application) • Source code management (Github, etc) • Pipeline automation tool (GoCD, Jenkins, etc) • Configuration Management (ansible , terraform) • Docker + Docker registry(docker hub, on prem registry) • Kubernetes (docker swarm, azure, etc)
17. Scaffold project (git, provision in the CD tool GoCD)
18. Source code management (Github)
19. Pipeline automation tool (GoCD)
20. Pipeline as a file
21. Deployment Pipeline Model • Write a code + tests • Setup Configuration in your SCM (versioning, dependencies) • Build + publish • Development env configuration • Staging env configuration • Production env configuration
22. Improvements • Packaging and dependency management in python is not so perfect • “Perfect” Continuous Delivery python tool, both GoCD and Jenkins2.0 are not so great: agents disconnects, tons of plugins, security flows • Every application is different has different components and architecture. Approach: - create a working “skeleton” for your team - things can be reused (images, tests, scripts) - provide standards (12 factor app rules, security)
23. Summary • Everyone can deploy at anytime • You build it, you run it • Moving Forward Faster • Automated better than manual
24. Thank you