Configuration Management

Configuration Management (Ansible, CloudFormation)
Starter Guide. Why Ansible and AWS CloudFormation.

Transcript

1. Configuration Management (Ansible, CloudFormation) A guide by Otieno Calvine
2. What we will cover ➢ Introduction to Ansible ➢ Inventory Files ➢ Playbooks ➢ Variables ➢ Conditionals ➢ Loops ➢ Roles ➢ Hands-On
3. Introduction to ANSIBLE Ansible is a powerful IT automation tool. It’s simple enough for everyone in IT yet very powerful to automate even the most complex deployment. It makes your application and systems easier to deploy. It supports configuration management with examples as below: Configuration management Security Performance Application Deployment Provisioning servers Continuous delivery
4. Simple Agentles Powerful Coding skills Time Maintenance Script Ansible
5. Why Ansible! Why Use it ➢ It is a free open source application ➢ Agent-less – No need for agent installation and management ➢ Python/yaml based ➢ Highly flexible and configuration management of systems. ➢ Large number of ready to use modules for system management ➢ Custom modules can be added if needed ➢ Configuration roll-back in case of error ➢ Simple and human readable ➢ Self documenting
7. Redhat or CentOS – sudo yum install ansible Ansible - Install Fedora – sudo dnf install ansible Ubuntu – sudo apt-get install ansible PIP – sudo pip install ansible Additional Options: • Install from source on GIT • Build RPM yourself https://docs.ansible.com/ansible/latest/installation_guide/ Control Machine: Linux Only
8. Inventory ➢ Ansible can work with one or multiple files in your infrastructure at the same time. ➢ In order to do this, ansible need to establish connectivity to those servers. ➢ This is done using SSH for Linux and PowerShell Remoting for windows. ➢ This is what makes ansible agentless. ➢ Information about these target machines is stored in an inventory file. ➢ If you don't create an inventory file, Ansible uses an inventory file located at etc/ansible/hosts. ➢ The inventory file is an INI-like format. ➢ It’s simply a number of servers listed one after the other. mail.example.com [webservers] foo.example.com bar.example.com [dbservers] one.example.com two.example.com three.example.com
9. More on Inventory Files # Web Servers web1 ansible_host=server1.company.com ansible_connection=ssh ansible_user=root ansible_ssh_pass=Password123! web2 ansible_host=server2.company.com ansible_connection=ssh ansible_user=root ansible_ssh_pass=Password123! web3 ansible_host=server3.company.com ansible_connection=ssh ansible_user=root ansible_ssh_pass=Password123! # Database Servers db1 ansible_host=server4.company.com ansible_connection=winrm ansible_user=administrator ansible_password=Password123! Inventory Parameters: ➢ ansible_connection – ssh/winrm/localhost ➢ ansible_port – 22/5986 ➢ ansible_user – root/administrator ➢ ansible_ssh_pass - Password Security: Ansible Vault
10. Playbooks ➢ Are ansible orchestration language. It is in playbooks that we define what we want ansible to do. ➢ It is a set of instructions you provide ansible to work its magic. ➢ It can be a as simple as running a series of commands on different servers in as sequence and restarting those servers in particular order or it could be as complex as deploying 100s of VMs in a public or private cloud infrastructure, provision storage to VMs, setting up their network, cluster configuration, configuring applications on them such as web servers or DbB servers etc. ➢ Playbooks are written in YAML file. It is a single YAML file containing set of plays. ➢ Plays - Define a set of activities (tasks) tun be run on hosts ➢ Task - An action to be performed on the host- ❖ Execute a command ❖ Run a script ❖ Install a package ❖ Shutdown/Restart a server
11. A simple playbook file playbook.yml - name: Play 1 hosts: localhost tasks: - name: Execute command ‘date’ command: date - name: Execute script on server script: test_script.sh - name: Install httpd service yum: name: httpd state: present - name: Start web server service: name: httpd state: started How to execute Ansible Playbook ➢ ansible-playbook <playbook file name>
12. Ways of executing Ansible Command ➢ Ansible Command ➢ ansible-playbooks command We use ansible command to perform one-off tasks such as test connectivity between ansible controller and target machines or to run a command like to shutdown set of servers. ➢ ansible all -m ping ➢ ansible all -a “/bin/echo hello”
13. Modules ➢ Different actions run by tasks. ◦ Command ◦ Script ◦ Yum ◦ service ➢ There are 100s of the modules available out of the box. Information about these modules are available in the Ansible Documentation website or you can simply run ansible-doc -L System - Are actions to be performed on the system level such as modifying users and groups on the system, modifying iptables, firewall configuration etc. Commands - Are used to execute commands or a script on a host Files - Help work with files. For example, use the ACL module to set and retrieve ACL information on files. Database - Use to work with databases like MongoDb, MySQL, PostgresQL to add or remove databases or modify database configurations. Cloud - A vast collection of modules for various cloud providers like Amazon, Azure, Docker, Google. Windows - Help use ansible in a Window environment. Some of these are win_copy to copy files, win_command to execute a command in Windows machine.
14. Modules - Continuation Playbook.yml - name: Start Services in order hosts: localhost tasks: - name: Start the database service service: name=postgresql state=started - name: Start the httpd service service: name=httpd state=started - name: Start the nginx service service: name: nginx state: started playbook.yml - name: Start Services in order hosts: localhost tasks: - name: Start the database service service: name: postgresql state: started
15. Ansible Variables ➢ Variables stores information that varies with each host. ◦ ansible_host ◦ ansible_connection ◦ ansible_ssh_pass ➢ Are examples of variables ➢ We can also define variables in playbooks ➢ We can also have variables defined in a separate ➢ File dedicated for variable. Playbook.yml - name: Add DNS server to resolv.conf hosts: localhost tasks: - lineinfile: path: /etc/resolv.conf line: 'nameserver 10.1.250.10' Playbook.yml - name: Add DNS server to resolv.conf hosts: localhost vars: dns_server: 10.1.250.10 tasks: - lineinfile: path: /etc/resolv.conf line: 'nameserver {{ dns_server }}'
16. Ansible Variables - Continuation - name: Set Firewall Configurations hosts: web tasks: - firewalld: service: https permanent: true state: enabled - firewalld: port:{{ http_port }}’/tcp permanent: true state: disabled #Sample Inventory File web http_port= snmp_port= inter_ip_range= #Sample variable File – web.yml http_port: 8081 snmp_port: 161-162 inter_ip_range: 192.0.2.0 {{ }} Jinja2 Templating source: {{ inter_ip_range }} - wrong source: ‘{{ inter_ip_range }}’ - correct source: SomeThing{{ inter_ip_range }}SomeThing - correct
17. Conditionals --- - name: Install NGINX hosts: debian_hosts tasks: - name: Install NGINX on Debian apt: name: nginx state: present --- - name: Install NGINX hosts: debian_hosts tasks: - name: Install NGINX on Redhat yum: name: nginx state: present
18. Conditional - When --- - name: Install NGINX hosts: debian_hosts tasks: - name: Install NGINX on Debian apt: name: nginx state: present when: nsible_os_family == “Debian” - name: Install NGINX on Redhat yum: name: nginx state: present when: nsible_os_family == “Redhat” --- - name: Install NGINX hosts: debian_hosts tasks: - name: Install NGINX on Debian apt: name: nginx state: present when: nsible_os_family == “Debian” - name: Install NGINX on Redhat yum: name: nginx state: present when: nsible_os_family == “Redhat” or when: nsible_os_family == “SUSE”
19. Conditional - When and operator OR --- - name: Install NGINX hosts: debian_hosts tasks: - name: Install NGINX on Debian apt: name: nginx state: present when: nsible_os_family == “Debian” - name: Install NGINX on Redhat yum: name: nginx state: present when: nsible_os_family == “Redhat” --- - name: Install NGINX hosts: debian_hosts tasks: - name: Install NGINX on Debian apt: name: nginx state: present when: nsible_os_family == “Debian” - name: Install NGINX on Redhat yum: name: nginx state: present when: nsible_os_family == “Redhat” or when: nsible_os_family == “SUSE”
20. Conditional - When and operator AND --- - name: Install NGINX hosts: debian_hosts tasks: - name: Install NGINX on Debian apt: name: nginx state: present when: nsible_os_family == “Debian” and when: ansible_distribution_version == “16.04” - name: Install NGINX on Redhat yum: name: nginx state: present when: nsible_os_family == “Redhat” or when: nsible_os_family == “SUSE”
21. Conditionals in Loops --- - name: Install Softwares hosts: all vars: packages: - name: nginx required: True - name: mysql required : True - name: apache required : False Task: - name: Install “{{ item.name }}” on Debian apt: name: “{{ item.name }}” state: present when: name: item.required == True loop: "{{ packages }}"
22. Loops - name: Create users hosts: localhost tasks: - user: name=calvine state=present - user: name=charles state=present - user: name=josephat state=present - user: name=kenneth state=present - user: name=willard state=present - user: name=mike state=present - user: name=winnie state=present - user: name=dexter state=present - user: name=sheliza state=present - name: Create users hosts: localhost tasks: - users: name= ‘{{ item }}’ state=present loop: - calvine - charles - josephat - kenneth - willard - winnie - sheliza
23. Loops: with_ * - name: Create users hosts: localhost tasks: - users: name= ‘{{ item }}’ state=present with_items: - calvine - charles - josephat - kenneth - willard - winnie - sheliza
24. Loops: with_ * - Continuation - name: Get from multiple URLs hosts: localhost tasks: - debug: var=item with_url: - “https://site1.com/get-servers” - “https://site2.com/get-servers” - “https://site3.com/get-servers” - name: Check multiple mongodbs hosts: localhost tasks: - debug: msg=“DB={{ item.database }} PID={{ item.pid}}” with_mongodb: - database: dev connection_string: “mongodb://dev.mongo/” - database: prod connection_string: “mongodb://prod.mongo/”
25. Roles - name: Install and Configure MySQL hosts: db-server tasks: - name: Install Prerequisites yum: name=pre-req-packages state=present - name: Install MySQL Packages yum: name=mysql state=present - name: Start MySQL Service service: name=mysql state=started - name: Configure Database mysql_db: name=db1 state=present This is a simple playbook ➢ Installing Prerequisites ➢ Installing mysql packages ➢ Configuring mysql service ➢ Configuring database and users
26. Roles - Continuation - name: Install and Configure MySQL hosts: db-server tasks: roles: - mysql tasks: - name: Install Prerequisites yum: name=pre-req-packages state=present - name: Install MySQL Packages yum: name=mysql state=present - name: Start MySQL Service service: name=mysql state=started - name: Configure Database mysql_db: name=db1 state=present
27. Roles - Continuation Why use roles ➢ Reuse - The primary purpose of roles is make your work reusable. Be it for other tasks or other projects within your organisation. ➢ Organize - Roles also help in organizing your code within ansible Vars mysql_packages: - mysql - mysql-server db_config: db_name: db1 Defaults mysql_user_name: root mysql_user_password: root handlers templates
28. Roles - Continuation Why use roles ➢ Share - Roles also help in sharing your code with others in the Ansible Community - Ansible Galaxy is one such community where you can find thousands of roles for almost any task you can think of. How to get started with Roles ➢ Create the directory structure required for a role using Ansible Galaxy Tool. Use the to ansible-galaxy init command to create a skeleton for you. ➢ This command will initialize and create a directory structure.
29. Roles - Continuation Using your roles within your playbook. There are different ways to do that. ➢ Create a directory within called roles within your playbooks folder and move the roles you created under it. When the playbook runs, Ansible looks for a role named, for example, mysql under the roles directory. ➢ You can move the role to a common directory designated for roles on your system at /etc/ansible/roles location. It’s the default location where ansible searches for roles if it can't be found in your playbook directory. This is defined in the ansible configuration file as roles path. /etc/ansible/ansible.cfg roles_path = /etc/ansible/roles
30. Roles - Continuation Find and use an existing role in your playbook ➢ You can search from Ansible Galaxy UI ➢ From command line interface using the ansible-galaxy search command ansible-galaxy search mysql ➢ To use a role, run the ansible-galaxy install command with the name of the role. ansible-galaxy install <role_name> ➢ To view the list of roles currently installed run the ansible-galaxy list command ansible-galaxy list ➢ To view the location where roles would be installed run the ansible-config dump | grep ROLE. ansible-config dump | grep ROLE ➢ While installing the roles you may use the “-p” option to install it in the current location under roles. ansible-galaxy install <role_name> –p ./roles
31. HANDS-ON
32. AWS CLOUDFORMATION Managing your infrastructure as a code ➢ An easy way to create and manage a collection of AWS resources. ➢ Allows orderly and predictable provisioning and updating of resources. ➢ Allows you to version control your AWS infrastructure. ➢ Deploy and update stacks using console, command line or API. ➢ You only pay for resources you create. CloudFormation is a declarative way of outlining your AWS Infrastructure, for any resources (most of them are supported)
33. Benefits of CloudFormation ➢ Infrastructure as code ➢ Cost ➢ Productivity ➢ Separation of concern ➢ Don't reinvent the wheel
34. How CloudFormation Works ➢ Templates have to be uploaded in S3 and then referenced in CloudFormation ➢ To update a template, we can’t edit previous ones. We have to re-upload a new version of the template to AWS ➢ Stacks are identified by a name ➢ Deleting a stack deletes every single artifact that was created by CloudFormation.
35. Deploying CloudFormation Templates Manual way: ➢ Editing templates in the CloudFormation Designer ➢ Using the console to input parameters, etc Automated way: ➢ Editing templates in a YAML/JSON file ➢ Using the AWS CLI (Command Line Interface) to deploy the templates ➢ Recommended way when you fully want to automate your flow
36. CloudFormation Building Blocks Templates components ➢ Resources: Your AWS resources declared in the template (MANDATORY) ➢ Parameters: The dynamic inputs for your template ➢ Mappings: The static variables for your template ➢ Outputs: References to what has been created ➢ Conditionals: List of conditions to perform resource creation ➢ Metadata Templates helpers: ➢ References ➢ Functions
37. High Level Template structure { "Description": "A text description for the template usage", "Parameters": { // A set of inputs used to customize the template per deployment }, "Resources": { // A set of AWS resources and relationships between them }, "Outputs": { // A set of values to be made visible to the stack creator }, "AWSTemplateFormatVersion": "2010- ‐09- ‐09" }
38. High Level Template structure { "Description": "A text description for the template usage", "Parameters": { // A set of inputs used to customize the template per deployment }, "Resources": { // A set of AWS resources and relationships between them }, "Outputs": { // A set of values to be made visible to the stack creator }, "AWSTemplateFormatVersion": "2010- ‐09- ‐09" }
39. Parameters ➢ They are optional and used to customize your template. ➢ Parameters enable you to input custom values to your template. This can be done each time you create or update a stack. Concept: Pseudo Parameters. Predefined by AWS CloudFormation
40. A simple template that creates an EC2 Instance JSON File format
41. A simple template that creates an EC2 Instance YAML File format
42. Resources ➢ Declares the required resources that you want to include in the stack. These can be: ◦ Amazon Elastic Compute Cloud(EC2) instance ◦ Amazon Simple Storage Service (S3) bucket etc ➢ All the resources can be found here AWS Resources
43. Parameters ➢ They are optional and used to customize your template. ➢ Parameters enable you to input custom values to your template. This can be done each time you create or update a stack. Concept: Pseudo Parameters. Predefined by AWS CloudFormation
44. Mappings ➢ Are fixed variables within your CloudFormation templates. ➢ All the values are hardcoded within AWS CloudFormation template. The following example shows a Mappings section with a map RegionMap, which contains five keys that map to name-value pairs containing single string values. The keys are region names. Each name-value pair is the AMI ID for the AMI in the region represented by the key
45. Mappings- Accessing Map values ➢ We use Fn::FindInMap to return a named value from a specific key. !FindInMap [ MapName, TopLevelKey, SecondLevelKey ]
46. Outputs ➢ Declares optional outputs values that we can import into other stacks that is if you export them first. ➢ You can also view the outputs in the AWS Console or in using the AWS CLI. ➢ It's the best way to perform collaboration across stack. NOTE: You can't delete a CloudFormation Stack if its outputs are being referenced by another CloudFormation stack
47. Conditions ➢ Define the circumstances under which entities are created or configured. ➢ Conditions can be whatever you want them to be, but common ones are: ◦ Environment (dev / test / prod) ◦ AWS Region ◦ Any parameter value Attaching a security group to an EC2 instance only in our production environment. This takes only one parameter for the environment name and a condition to test whether it is production
48. CloudFormation Must Know Intrinsic Functions ➢ Ref ➢ Fn::GetAtt ➢ Fn::FindInMap ➢ Fn::ImportValue ➢ Fn::Join ➢ Fn::Sub ➢ Condition Functions (Fn::If, Fn::Not, Fn::Equals, etc...)
49. CloudFormation Rollbacks ➢ Stack Creation Fails: ◦ Default: everything rolls back (gets deleted). We can look at the log ◦ Option to disable rollback and troubleshoot what happened ➢ Stack Update Fails: ◦ The stack automatically rolls back to the previous known working state ◦ Ability to see in the log what happened and error messages
50. ChangeSets When you update a stack, you need to know what changes before it happens for greater confidence
51. Nested Stacks ➢ Stacks created as part of other stacks ➢ They allow you to isolate repeated patterns / common components in separate stacks and call them from other stacks Example: ◦ Load Balancer configuration that is reused ◦ Security Group that is reused ➢ Nested stacks are considered best practice ➢ To update a nested stack, always update the parent (root stack)
52. CloudFormation - StackSets ➢ A stack set lets you create stacks in AWS accounts across regions by using a single AWS CloudFormation template. ➢ Administrator account to create StackSets ➢ Trusted accounts to create, update, delete stack instances from StackSets ➢ When you update a stack set, all associated stack instances are updated throughout all accounts and regions
53. THE END THANK YOU!