DevOps-Andela

A talk on Development Operations given at Andela in Nairobi in February 2016 by @jmwai

Transcript

1. DevOps Andela Nairobi 2016
2. Agenda • What is DevOps • Operations • Monitoring • Logging and Metrics • Benefits of adopting a DevOps approach • Q&A
3. What is DevOps People have come up with different definitions of DevOps Wikipedia - Development methodology with an emphasis on collaboration, integration, automation. Brings together software development, QA and IT operations. Techcrunch - A combination of things that enabled Automated and Continuous deployment. With modern applications increasingly running in the cloud, much of infrastructure is now part of the code.
4. What is DevOps • Software Development ◦ Code ◦ Test ◦ Deploy • Quality Assurance ◦ Testing ◦ Continuous Integrations • Operations ◦ Orchestration ◦ Logging ◦ Monitoring
5. Operations Automated configuration and deployment VM’s and Containers Monitoring Logging
6. Common Infrastructure Servers
7. 1. DNS Resolvers 2. Web Servers 3. Database Servers 4. Caching Servers 5. Storage Servers 6. Load Balancers 7. Queueing Servers 8. CDN 9. NAT 10. Test 11. Build 12. ……..
8. Automated Configuration & Deployment Automating the process of application deployment, configuration management and Continuous Delivery will save your organisation time and money. Tools that come in handy: ★ Ansible ★ Capistrano ★ Fabric ★ SaltStack ★ Chef ★ Puppet
9. Automated Configuration & Deployment SaltStack • Scalable and flexible configuration management software for event-driven infrastructure automation. • Written in Python. You can extend it using Python • You have a Salt-Master which hosts all the configurations policies • Salt-Minions are the nodes in your infrastructure that you want managed. • Salt uses ZeroMQ to push configs between Master and minions • https://docs.saltstack.com
10. Automated Configuration & Deployment Capistrano • Written in Ruby • Popular in the Rails community • Can be used with other languages as well • Organises operative directions in files known as recipes • Not just userful with deployment, you can use it to build servers, install applications, maintenance and monitoring • Great documentation at http://capistranorb.com/#
11. Automated Configuration & Deployment Chef • Powerful configuration management system that can be used to programmatically control your infrastructure environment in a predictable manner • Also uses ruby and stores config details in files called recipes. • Chef Server: Central location for your configuration recipes, cookbooks, and node and workstation definitions. Other machines in your infrastructure use the chef server for deployment configuration. • Chef Nodes: Chef nodes are the deployment targets that are configured by Chef. • Chef Workstations: Where Chef configuration details are created or edited. The configs are then pushed to the Chef server, for deployment to any node
12. Automated Configuration & Deployment Puppet • Open Source automation software from Puppet Labs • There is an enterprise versions as well • Describes machine configuration in a declarative language. • Config code is stored in files called resource declarations • Declarations are organised into manifests. • A collection of manifests and data are known as modules.
13. Automated Configuration & Deployment Fabric • Python Library for command-line streamlining use of SSH. • Provide a host of operations for executing local and remote SSH commands. • You write your tasks as python functions. Store them in a fabfile.py file and fabric will execute them on demand. • It’s just a python library on top of ssh that allows you to store repetitive ssh tasks in python files. • Different from others in that it uses the push methodology while chef,ansible loads your configs and executes on the server.
14. Automated Configuration & Deployment Ansible • According to the creators, Ansible is the simplest way to automate apps and IT infrastructure • Uses YAML syntax. • Configuration details are organised in roles with each roles containing it’s own files, tasks, handlers, vars, templates. • Playbooks contain roles and target hosts. • Plenty of free tested roles available on https://galaxy.ansible.com • Recently acquired by Redhat
15. Ansible Example - name: Install Nginx sudo: yes apt: pkg=nginx state=installed update_cache=yes - name: Disable the default site sudo: yes file: path=/etc/nginx/sites-enabled/default state=absent - name: Install Nginx Configuration File. sudo: yes copy: src={{ project_name }} dest=/etc/nginx/sites-enabled/{{ project_name }} notify: Reload Nginx
16. VM and Containers Virtual Machines - Emulation of an entire operating system. Runs a full copy of the OS and virtual copy of the hardware needed. Heavy use of RAM, CPU... Linux Containers (LXC) - OS-level virtualization allowing you to run multiple Linux instances(Containers) in a single host. Linux Container allows multiple isolation and prioritization of CPU, Memory, I/O, Network without the need to launch virtual machines. Container managers allow you to run multiple isolated linux instances on a single host.; • Docker - By far the most popular container format • Rocket - Created by CoreOs • Odin...
17. Docker • Automate deployment of applications in Linux containers. • Installation On Ubuntu is a simple as apt-get install docker.io. • Docker Image - Packages your application configuration details; Directories, software dependencies, your code... • Docker Container - Instance of an image. • Dockerfile - A file containing declarations on automating the image building process • Docker Compose - Organises your containers into logical units enabling them to communicate with each other. • Docker Hub - Docker container hosting
18. #Docker Pombola Container Example FROM i686/ubuntu:14.04 MAINTAINER James Mwai <xyz@gmail.com> RUN apt-get update RUN apt-get install -y python-software-properties # + all other dependencies RUN pip install --upgrade pip RUN pip install --upgrade virtualenv RUN mkdir pombola_root && cd pombola_root RUN git clone https://github.com/mysociety/pombola.git
19. Scaling Your Docker cluster Docker Swarm Kubernetes Mesos Fleet Amazon ECS
20. Docker Swarm Native container clustering tool created by Docker. It uses the standard Docker API Turns a groups of Docker containers into a single virtual Docker engine You have a Swarm Manager which acts as the Master with each host running the Swarm Agent. Can scale up to 1000 nodes running 50000
21. Kubernetes Open source orchestration system for Docker containers. Created by Google leveraging their internal experience using containers in production for the last decade. Containers are organised into logical units called pods. Kubernetes schedules your pods and grows them in your cluster Still under active development
22. Apache Mesos Open source clustering engine started at UC Berkeley. Designed to scale to very large clusters running on thousands of hosts with high availability and resilience. Used by large companies with like Twitter, eBay, Airbnb.
23. Fleet Low-level cluster management tool from CoreOs. Allows you to deploy docker containers on arbitrary hosts on a cluster. Built on top of Systemd Whereas Systemd is a system and service for a single linux host, fleet extends this to a cluster of machines. Allows you to treat your cluster of machines as a single host.
24. Amazon ECS Elastic Compute Container Service. Highly scalable, fast, container management service for running Docker containers on Amazon EC2 Lets you launch, stop, schedule, docker containers with simple API calls. Containers are grouped into clusters deployed on a fleet of EC2 instances.
25. Other Container Concepts • Open Container Initiative • CoreOS • Google Container Engine • Rocket
26. Monitoring • Graphite - Can store logs and render graphs of this data on demand • New Relic - Collect and display data on your apps, infrastructure. Real time updates. • Cloud Watch - Monitoring as a service for AWS resources. Can monitor EC2 servers, RDS, DynamoDB... • Nagios - Monitor your entire infrastructure - Systems, networks, infrastructure. nagios.org
27. Logging and Metrics • Sentry - Real Time exception tracking for webapps. Started in the Django community. Now a paid service. • LogStash - Collect and analyze logs from various sources. • Kibana - Collect, analyze, visualize, search your logs. • Datadog - Collect and visualize various performance metrics from your apps.
28. Benefits of adopting DevOps approach • Improved deployment frequency. • Lower failure rate • Faster recovery in case of failure • Faster fixing of problems • Faster delivery of features • More stable operating environments • More time available to add value (rather than fix/maintain)
29. Are you a modern web dev? Can you make a build in one step? Do you make daily builds? Do you use configuration management tools to automate infrastructure? Is the development setup documented and automated? Can you rollout and rollback deployments in one step? Can applications be deployed to a new server setup without changes to the code?
30. @jmwai