Pro Yearly is on sale from $80 to $50! »

2017 - Natalie Serebryakov - Continuous Deployment

Db2ee812bdc6fd057f8f4209c08b6f63?s=47 PyBay
August 13, 2017

2017 - Natalie Serebryakov - Continuous Deployment

Description
The presentation will be helpful for engineers, who developed an application in Python, or library or microservice. The presentation will cover the practices automating continuous integration, continuous delivery, and continuous deployment with Python ( examples: Jenkins and Gocd).

Abstract
Continuous Deployment is a process to ensure that code can be rapidly and safely deployed to production by delivering every change to a production-like environment and ensuring business applications and services(microservices) function as expected.

Bio
My name is Natalie; I have a Networking and Security background but recently decided to challenge myself in different space. That's why currently I work at Vevo as Tools Engineer in coolest DevOps team in San Francisco. Where part of my job includes building and automating various tools that help our development team be more productive and happy. I have an MS in Compute Since degree. For my first job, I had to do a lot of scripting for configuring and managing thousands of switches, routers and other networking devices in our company testing lab. After using sed, awk, Tcl I discovered and learned Python. It was straightforward to pick up and worked well since then Python became my language of preference. I also like Python for its big community.

In my spare time, I like to spend time with my pup Chai, bake and blog about baking.

Thank you, organizers, for giving me and people the opportunity to speak and this conference and share our knowledge with community .

Db2ee812bdc6fd057f8f4209c08b6f63?s=128

PyBay

August 13, 2017
Tweet

Transcript

  1. Continuous Deployment with Python Natalie Serebryakova PyBay2017

  2. About me Software Engineer (Tools) DevOps Team, Vevo LLC Love

    to get hands dirty on automating tasks and processes natalieserebry@gmail.com
  3. Agenda • What is Continuous Delivery • Why and how

    CD should be implemented • Delivery pipeline • Improvements
  4. What is Continuous Delivery ? Continuous delivery (CD) is a

    capability to continually deploy software to production as frequent as needed. The goal of Continuous Delivery is to enable a constant flow of changes into production via an automated software production line.
  5. Why should we start doing this ? • Improve speed

    to market • Ensured quality • Deploying code is seamless, no longer strenuous • Improves your disaster recovery/business continuity • Improve business agility
  6. Implementing Continuous Delivery In our Continuous Delivery Pattern, we implemented

    two architectural attributes: • testability (“passed” on every environment) • deployability ( low-risk to “build” on every environment) Designing for testability and deployability starts with ensuring our services and modules are composed of loosely-coupled, well-encapsulated components follow the principles of SOLID design
  7. SOLID (object-oriented design) In computer programming, the term SOLID is

    a mnemonic acronym for five design principles intended to make software designs more understandable, flexible and maintainable. (Wiki)
  8. Why to automate? • It’s easy with python, many modules

    available: gocd, jenkinsapi, python-jenkins, docker-py, pykube, etc • save time, stop repeat the same things (DRY) • the complete delivery pipeline is in git: you have predictable recovery because, you know what is happening • machines do it just better • provide tools Development team maintain and can refactor their deployment • no UI (no “you just have to click on”....)
  9. How to automate. Real world example The goal in allow

    Development Team run the single command to: • Scaffold new project (git, provision in the CD tool GoCD, Jenkins) • Checkout it from the source code • Run automated tests • Deploy to every environment
  10. I have working python code, how do I start now?

    Database (PostrgreSQL) App (Flask Gunicorn) Static Assets Webserver (NGINX) TCP Socket Browser HTTP Layout of a non-trivial Flask application Single Server
  11. I have working python code, how do I start now?

    Browser Running many instances of the app, talking to a single DB Application Server 1 App (Flask Gunicorn) Application Server 2 App (Flask Gunicorn) Application Server 3 App (Flask Gunicorn) Static Assets Webserver (NGINX) HTTP Database (PostrgreSQL) TCP Database Server Load Balancer HTTP HTTP HTTP TCP TCP
  12. The application running on an abstract collection of resources in

    production environment Browser primitives such as persistent storage and load balancers that are common to any non-trivial Node 2 App (Flask Gunicorn) App (Flask Gunicorn) Node 3 App (Flask Gunicorn) Content Delivery Network (Static Assets) Load balancer HTTP Database (PostrgreSQL) TCP Node 1 HTTP HTTP HTTP TCP TCP Persistent Disk HTTP the primary pieces we care about (the application components)
  13. Architecture using Deployment pipeline The key pattern introduced in Continuous

    Delivery is the deployment pipeline Deploy Staging Deploy Production Deploy Development Build Test
  14. The pipeline breaks down the software delivery process into stages.

    Each stage is aimed at verifying the quality of new features from a different angle to validate the new functionality and prevent errors from affecting your users. The pipeline should provide feedback to the team and visibility into the flow of changes to everyone involved in delivering the new feature/s. The Continuous Delivery pipeline
  15. Proposed solution from the slide 9 Development Team should run

    the single command • Scaffold new project (git, provision in the CD tool GoCD) • Checkout it from the source code • Run automated tests • Deploy to every environment
  16. Components for automated deploys • Python package (application) • Source

    code management (Github, etc) • Pipeline automation tool (GoCD, Jenkins, etc) • Configuration Management (ansible , terraform) • Docker + Docker registry(docker hub, on prem registry) • Kubernetes (docker swarm, azure, etc)
  17. Scaffold project (git, provision in the CD tool GoCD)

  18. Source code management (Github)

  19. Pipeline automation tool (GoCD)

  20. Pipeline as a file

  21. Deployment Pipeline Model • Write a code + tests •

    Setup Configuration in your SCM (versioning, dependencies) • Build + publish • Development env configuration • Staging env configuration • Production env configuration
  22. Improvements • Packaging and dependency management in python is not

    so perfect • “Perfect” Continuous Delivery python tool, both GoCD and Jenkins2.0 are not so great: agents disconnects, tons of plugins, security flows • Every application is different has different components and architecture. Approach: - create a working “skeleton” for your team - things can be reused (images, tests, scripts) - provide standards (12 factor app rules, security)
  23. Summary • Everyone can deploy at anytime • You build

    it, you run it • Moving Forward Faster • Automated better than manual
  24. Thank you